Sep 05 02:18:16.317743 ip-10-0-143-128 systemd[1]: Starting Kubernetes Kubelet...
Sep 05 02:18:16.729231 ip-10-0-143-128 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:18:16.729231 ip-10-0-143-128 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 05 02:18:16.729231 ip-10-0-143-128 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:18:16.729231 ip-10-0-143-128 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 05 02:18:16.730218 ip-10-0-143-128 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:18:16.732694 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.732527    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 05 02:18:16.737638 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737622    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:16.737638 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737637    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737642    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737646    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737649    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737652    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737655    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737658    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737661    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737664    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737666    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737669    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737672    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737674    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737677    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737680    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737683    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737685    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737688    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737698    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737701    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:16.737725 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737704    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737706    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737709    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737712    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737714    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737717    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737720    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737722    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737725    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737727    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737730    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737733    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737736    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737739    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737742    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737744    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737747    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737751    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737755    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737757    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:16.738269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737759    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737762    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737765    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737767    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737770    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737772    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737776    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737779    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737782    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737785    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737787    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737791    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737794    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737796    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737800    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737804    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737806    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737809    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:16.738817 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737812    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737814    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737817    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737819    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737822    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737825    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737828    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737831    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737833    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737836    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737838    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737841    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737843    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737846    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737848    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737851    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737854    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737856    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737859    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737861    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737864    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:16.739256 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737867    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737869    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737872    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737874    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737878    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737880    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737884    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737887    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737890    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737892    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737895    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737897    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737900    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737902    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737905    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737907    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737910    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737913    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.737916    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738625    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:16.739831 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738632    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738638    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738642    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738646    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738649    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738652    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738655    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738658    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738660    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738663    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738665    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738669    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738671    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738675    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738678    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738681    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738683    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738686    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738689    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738692    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:16.740311 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738694    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738697    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738700    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738702    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738705    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738707    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738710    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738714    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738718    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738720    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738723    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738726    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738729    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738732    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738734    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738738    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738740    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738743    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738746    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:16.740865 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738748    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738751    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738754    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738757    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738761    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738763    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738766    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738769    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738772    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738774    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738777    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738780    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738782    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738785    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738788    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738791    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738793    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738795    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738798    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738801    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:16.741325 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738804    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738806    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738809    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738811    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738814    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738816    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738819    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738822    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738825    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738827    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738830    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738834    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738837    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738839    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738842    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738845    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738848    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738850    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738853    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738856    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:16.741832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738859    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738862    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738865    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738868    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738871    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738874    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738876    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738879    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738882    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738884    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738887    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738890    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738893    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738895    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738899    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738903    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738906    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738909    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.738912    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:16.742318 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739288    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739293    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739296    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739300    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739303    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739306    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739309    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739311    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739314    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739317    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739319    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739321    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739324    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739327    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739330    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739333    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739336    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739338    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739341    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739344    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:16.742797 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739346    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739349    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739351    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739354    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739357    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739359    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739362    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739365    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739367    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739370    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739372    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739375    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739378    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739380    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739383    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739385    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739388    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739390    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739393    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739395    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:16.743286 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739398    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739400    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739403    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739405    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739407    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739410    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739416    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739418    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739423    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739425    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739428    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739430    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739433    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739450    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739452    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739455    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739457    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739460    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739463    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739466    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:16.743783 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739469    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739471    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739474    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739477    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739479    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739482    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739484    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739486    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739489    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739491    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739494    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739502    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739505    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739507    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739510    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739512    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739515    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739518    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739522    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739529    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:16.744268 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739533    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739536    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739538    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739543    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739545    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739548    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739550    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739553    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739555    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739558    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739562    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739566    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739569    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739572    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739574    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739577    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739579    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739582    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.739585    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:16.744796 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739654    2557 flags.go:64] FLAG: --address="0.0.0.0"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739689    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739696    2557 flags.go:64] FLAG: --anonymous-auth="true"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739700    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739704    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739707    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739711    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739715    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739718    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739721    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739725    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739728    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739731    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739736    2557 flags.go:64] FLAG: --cgroup-root=""
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739739    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739742    2557 flags.go:64] FLAG: --client-ca-file=""
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739745    2557 flags.go:64] FLAG: --cloud-provider="external"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739748    2557 flags.go:64] FLAG: --cluster-dns="[]"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739752    2557 flags.go:64] FLAG: --cluster-domain=""
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739755    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739758    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739763    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739767    2557 flags.go:64] FLAG: --container-log-max-files="5"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739771    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 05 02:18:16.745262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739773    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739776    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739779    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739782    2557 flags.go:64] FLAG: --contention-profiling="false"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739786    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739789    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739792    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739794    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739798    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739810    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739814    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739817    2557 flags.go:64] FLAG: --enable-load-reader="false"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739820    2557 flags.go:64] FLAG: --enable-server="true"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739822    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739827    2557 flags.go:64] FLAG: --event-burst="100"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739830    2557 flags.go:64] FLAG: --event-qps="50"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739833    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739836    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739839    2557 flags.go:64] FLAG: --eviction-hard=""
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739843    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739846    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739851    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739854    2557 flags.go:64] FLAG: --eviction-soft=""
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739857    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739860    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 05 02:18:16.745874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739863    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739866    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739869    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739872    2557 flags.go:64] FLAG: --fail-swap-on="true"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739876    2557 flags.go:64] FLAG: --feature-gates=""
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739879    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739882    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739885    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739888    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739891    2557 flags.go:64] FLAG: --healthz-port="10248"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739894    2557 flags.go:64] FLAG: --help="false"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739897    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-143-128.ec2.internal"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739900    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739903    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739906    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739909    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739913    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739915    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739918    2557 flags.go:64] FLAG: --image-service-endpoint=""
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739921    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739923    2557 flags.go:64] FLAG: --kube-api-burst="100"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739926    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739929    2557 flags.go:64] FLAG: --kube-api-qps="50"
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739932    2557 flags.go:64] FLAG: --kube-reserved=""
Sep 05 02:18:16.746557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739935    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739938    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739941    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739943    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739947    2557 flags.go:64] FLAG: --lock-file=""
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739950    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739953    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739957    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739962    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739964    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739967    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739970    2557 flags.go:64] FLAG: --logging-format="text"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739973    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739977    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739979    2557 flags.go:64] FLAG: --manifest-url=""
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739982    2557 flags.go:64] FLAG: --manifest-url-header=""
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739987    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739990    2557 flags.go:64] FLAG: --max-open-files="1000000"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739993    2557 flags.go:64] FLAG: --max-pods="110"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739996    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.739999    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740002    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740005    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740008    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740011    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 05 02:18:16.747150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740014    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740025    2557 flags.go:64] FLAG: --node-status-max-images="50"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740028    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740031    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740034    2557 flags.go:64] FLAG: --pod-cidr=""
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740037    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740043    2557 flags.go:64] FLAG: --pod-manifest-path=""
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740046    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740049    2557 flags.go:64] FLAG: --pods-per-core="0"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740052    2557 flags.go:64] FLAG: --port="10250"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740055    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740058    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0672869202c34424b"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740063    2557 flags.go:64] FLAG: --qos-reserved=""
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740066    2557 flags.go:64] FLAG: --read-only-port="10255"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740069    2557 flags.go:64] FLAG: --register-node="true"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740072    2557 flags.go:64] FLAG: --register-with-taints=""
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740076    2557 flags.go:64] FLAG: --registry-burst="10"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740079    2557 flags.go:64] FLAG: --registry-qps="5"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740082    2557 flags.go:64] FLAG: --reserved-cpus=""
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740084    2557 flags.go:64] FLAG: --reserved-memory=""
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740088    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740091    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740094    2557 flags.go:64] FLAG: --rotate-certificates="false"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740096    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740100    2557 flags.go:64] FLAG: --runonce="false"
Sep 05 02:18:16.747773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740103    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740106    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740108    2557 flags.go:64] FLAG: --seccomp-default="false"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740111    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740114    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740117    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740120    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740123    2557 flags.go:64] FLAG: --storage-driver-password="root"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740126    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740129    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740132    2557 flags.go:64] FLAG: --storage-driver-user="root"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740135    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740138    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740141    2557 flags.go:64] FLAG: --system-cgroups=""
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740143    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740149    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740152    2557 flags.go:64] FLAG: --tls-cert-file=""
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740155    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740158    2557 flags.go:64] FLAG: --tls-min-version=""
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740163    2557 flags.go:64] FLAG: --tls-private-key-file=""
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740165    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740168    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740172    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740175    2557 flags.go:64] FLAG: --v="2"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740179    2557 flags.go:64] FLAG: --version="false"
Sep 05 02:18:16.748378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740183    2557 flags.go:64] FLAG: --vmodule=""
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740187    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.740190    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740309    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740314    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740317    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740320    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740323    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740325    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740328    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740330    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740334    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740339    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740342    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740344    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740348    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740352    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740357    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740359    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:16.749002 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740362    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740365    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740368    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740371    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740374    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740377    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740379    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740383    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740386    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740388    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740391    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740394    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740397    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740399    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740402    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740404    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740407    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740409    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740411    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740414    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:16.749474 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740417    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740419    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740422    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740424    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740427    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740429    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740432    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740447    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740451    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740455    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740460    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740464    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740466    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740469    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740472    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740474    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740477    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740480    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740482    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:16.749992 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740485    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740489    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740493    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740496    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740499    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740503    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740506    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740509    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740512    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740515    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740517    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740520    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740524    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740527    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740529    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740532    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740535    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740537    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740540    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740542    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:16.750483 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740545    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740548    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740550    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740554    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740557    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740559    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740562    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740565    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740567    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740570    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740573    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740575    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740578    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740581    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740583    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740586    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740589    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740591    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740594    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740597    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:16.750969 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740599    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:16.751467 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740602    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:16.751467 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740604    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:16.751467 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.740607    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:16.751467 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.741195    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:18:16.751467 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.747938    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 05 02:18:16.751467 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.747952    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 05 02:18:16.751467 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.748883    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 05 02:18:16.751467 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.748891    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 05 02:18:16.751467 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748950    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:16.751467 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748955    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:16.751467 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748959    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:16.751467 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748963    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:16.751467 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748967    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:16.751467 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748970    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748974    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748977    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748979    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748982    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748985    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748988    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748991    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748993    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748996    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.748999    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749002    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749005    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749008    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749011    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749014    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749017    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749020    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749023    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749026    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:16.751841 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749029    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749031    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749034    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749037    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749040    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749043    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749047    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749051    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749054    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749056    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749059    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749062    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749065    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749067    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749070    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749073    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749075    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749078    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749080    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749083    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:16.752319 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749086    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749088    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749091    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749093    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749095    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749098    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749101    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749103    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749106    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749109    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749112    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749114    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749117    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749120    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749122    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749125    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749129    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749133    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749136    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:16.752830 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749139    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749142    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749144    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749147    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749150    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749153    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749155    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749158    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749161    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749163    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749166    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749168    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749171    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749173    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749176    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749178    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749181    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749183    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749186    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749188    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:16.753291 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749191    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749193    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749196    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749199    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749202    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749204    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749207    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749209    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749216    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749218    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749221    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749224    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749226    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749229    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749232    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:16.753868 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.749236    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749347    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749352    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749355    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749358    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749361    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749363    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749366    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749369    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749372    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749375    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749377    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749380    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749382    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749385    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749387    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749390    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749392    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749397    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:18:16.754247 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749400    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749403    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749406    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749409    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749411    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749414    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749417    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749420    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749422    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749425    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749427    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749430    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749432    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749445    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749448    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749451    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749453    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749456    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749458    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:18:16.754747 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749461    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749463    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749466    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749468    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749471    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749474    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749476    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749479    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749482    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749486    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749489    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749493    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749496    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749499    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749502    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749505    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749508    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749511    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749513    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749516    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:18:16.755210 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749519    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749521    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749524    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749527    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749530    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749532    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749535    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749537    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749540    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749542    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749545    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749547    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749550    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749553    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749555    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749557    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749560    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749563    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749566    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749568    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:18:16.755736 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749571    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749573    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749575    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749578    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749581    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749583    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749585    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749588    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749590    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749593    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749596    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749598    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749601    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749603    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749606    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749609    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749611    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749613    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749616    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749618    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:18:16.756219 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749621    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:18:16.756737 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:16.749623    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:18:16.756737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.749629    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:18:16.756737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.749725    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 05 02:18:16.756737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.753363    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 05 02:18:16.756737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.754300    2557 server.go:1019] "Starting client certificate rotation"
Sep 05 02:18:16.756737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.754465    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:18:16.756737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.754500    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:18:16.756737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.756674    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:18:16.760742 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.760720    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:18:16.774215 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.774198    2557 log.go:25] "Validated CRI v1 runtime API"
Sep 05 02:18:16.777039 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.777026    2557 log.go:25] "Validated CRI v1 image API"
Sep 05 02:18:16.777855 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.777838    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 05 02:18:16.781118 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.781096    2557 fs.go:135] Filesystem UUIDs: map[18cc8a23-b8e3-4f2c-a23f-97001754a3bd:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 82cade8c-807a-402e-b5ab-197465d48793:/dev/nvme0n1p3]
Sep 05 02:18:16.781183 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.781115    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 05 02:18:16.781629 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.781614    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:18:16.786887 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.786777    2557 manager.go:216] Machine: {Timestamp:2026-09-05 02:18:16.78533457 +0000 UTC m=+0.340490872 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3085578 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec261bf5449c7abd0cde12da83798df8 SystemUUID:ec261bf5-449c-7abd-0cde-12da83798df8 BootID:1870208f-ade9-49d9-866e-c807b5bd4957 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4d:ef:fb:ec:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4d:ef:fb:ec:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:9a:0c:ba:1b:f1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 05 02:18:16.786887 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.786886    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 05 02:18:16.787000 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.786962    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 05 02:18:16.790334 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.790305    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 05 02:18:16.790518 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.790336    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-128.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 05 02:18:16.791912 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.791899    2557 topology_manager.go:138] "Creating topology manager with none policy"
Sep 05 02:18:16.791976 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.791917    2557 container_manager_linux.go:309] "Creating device plugin manager"
Sep 05 02:18:16.791976 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.791930    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:18:16.794147 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.794134    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:18:16.795134 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.795122    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 05 02:18:16.798327 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.798313    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:18:16.798471 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.798460    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 05 02:18:16.798541 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.798498    2557 kubelet.go:486] "Attempting to sync node with API server"
Sep 05 02:18:16.798541 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.798513    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 05 02:18:16.798541 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.798528    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 05 02:18:16.798541 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.798541    2557 kubelet.go:398] "Adding apiserver pod source"
Sep 05 02:18:16.798693 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.798565    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 05 02:18:16.798928 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.798914    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:18:16.798987 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.798939    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:18:16.801414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.801394    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 05 02:18:16.803531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.803514    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-78c8h"
Sep 05 02:18:16.804795 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.804782    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 05 02:18:16.804839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.804812    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 05 02:18:16.806059 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.806048    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 05 02:18:16.806101 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.806064    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 05 02:18:16.806101 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.806071    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 05 02:18:16.806101 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.806077    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 05 02:18:16.806101 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.806085    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 05 02:18:16.806101 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.806091    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 05 02:18:16.806101 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.806103    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 05 02:18:16.806258 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.806109    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 05 02:18:16.806258 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.806116    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 05 02:18:16.806258 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.806122    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 05 02:18:16.806258 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.806130    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 05 02:18:16.806258 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.806135    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 05 02:18:16.806973 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.806963    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 05 02:18:16.806973 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.806973    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 05 02:18:16.811101 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.811077    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-128.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 05 02:18:16.811196 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:16.811099    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 05 02:18:16.811262 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:16.811194    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 05 02:18:16.811717 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.811702    2557 server.go:1268] "Started kubelet"
Sep 05 02:18:16.811817 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.811770    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 05 02:18:16.811872 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.811836    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 05 02:18:16.811922 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.811908    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 05 02:18:16.812625 ip-10-0-143-128 systemd[1]: Started Kubernetes Kubelet.
Sep 05 02:18:16.813975 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.813957    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 05 02:18:16.814142 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.814126    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-78c8h"
Sep 05 02:18:16.814369 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.814355    2557 server.go:310] "Adding debug handlers to kubelet server"
Sep 05 02:18:16.818521 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:16.816779    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-128.ec2.internal.18d24b35e1b608f2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-128.ec2.internal,UID:ip-10-0-143-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-128.ec2.internal,},FirstTimestamp:2026-09-05 02:18:16.811661554 +0000 UTC m=+0.366817856,LastTimestamp:2026-09-05 02:18:16.811661554 +0000 UTC m=+0.366817856,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-128.ec2.internal,}"
Sep 05 02:18:16.819715 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:16.819697    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 05 02:18:16.820219 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.820207    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 05 02:18:16.820688 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.820673    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 05 02:18:16.820856 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:16.820831    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-128.ec2.internal\" not found"
Sep 05 02:18:16.821287 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.821268    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 05 02:18:16.821287 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.821289    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 05 02:18:16.821421 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.821267    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 05 02:18:16.821421 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.821344    2557 factory.go:55] Registering systemd factory
Sep 05 02:18:16.821421 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.821360    2557 factory.go:223] Registration of the systemd container factory successfully
Sep 05 02:18:16.821421 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.821366    2557 reconstruct.go:97] "Volume reconstruction finished"
Sep 05 02:18:16.821421 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.821373    2557 reconciler.go:29] "Reconciler: start to sync state"
Sep 05 02:18:16.821655 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.821571    2557 factory.go:153] Registering CRI-O factory
Sep 05 02:18:16.821655 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.821581    2557 factory.go:223] Registration of the crio container factory successfully
Sep 05 02:18:16.821655 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.821620    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 05 02:18:16.821655 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.821635    2557 factory.go:103] Registering Raw factory
Sep 05 02:18:16.821655 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.821646    2557 manager.go:1196] Started watching for new ooms in manager
Sep 05 02:18:16.821935 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.821922    2557 manager.go:349] Starting recovery of all containers
Sep 05 02:18:16.831781 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.831613    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:16.832326 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.832310    2557 manager.go:354] Recovery completed
Sep 05 02:18:16.834754 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:16.834729    2557 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-128.ec2.internal\" not found" node="ip-10-0-143-128.ec2.internal"
Sep 05 02:18:16.838289 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.838275    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:16.839872 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.839857    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:16.839940 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.839894    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:16.839940 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.839908    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:16.840530 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.840515    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 05 02:18:16.840530 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.840529    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 05 02:18:16.840663 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.840531    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:16.840663 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.840548    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:18:16.841394 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.841331    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:16.841394 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.841352    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:16.841394 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.841365    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:16.841713 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.841699    2557 policy_none.go:49] "None policy: Start"
Sep 05 02:18:16.841767 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.841719    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 05 02:18:16.841767 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.841733    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 05 02:18:16.843044 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.843031    2557 policy_none.go:47] "Start"
Sep 05 02:18:16.873117 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.873103    2557 manager.go:337] "Starting Device Plugin manager"
Sep 05 02:18:16.884623 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:16.873135    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 05 02:18:16.884623 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.873145    2557 server.go:87] "Starting device plugin registration server"
Sep 05 02:18:16.884623 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.873467    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 05 02:18:16.884623 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.873479    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 05 02:18:16.884623 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.873573    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 05 02:18:16.884623 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.873650    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 05 02:18:16.884623 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.873658    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 05 02:18:16.884623 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:16.874179    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 05 02:18:16.884623 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:16.874227    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-128.ec2.internal\" not found"
Sep 05 02:18:16.973889 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.973857    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:16.974767 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.974750    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:16.974852 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.974778    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:16.974852 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.974787    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:16.974852 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.974813    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-128.ec2.internal"
Sep 05 02:18:16.982653 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.982580    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-128.ec2.internal"
Sep 05 02:18:16.982653 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:16.982601    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-128.ec2.internal\": node \"ip-10-0-143-128.ec2.internal\" not found"
Sep 05 02:18:16.982814 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.982798    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 05 02:18:16.984036 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.984016    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 05 02:18:16.984118 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.984061    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 05 02:18:16.984118 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.984081    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 05 02:18:16.984216 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:16.984161    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 05 02:18:16.986178 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:16.986159    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:17.028115 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:17.028090    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-128.ec2.internal\" not found"
Sep 05 02:18:17.085195 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.085164    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-128.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal"]
Sep 05 02:18:17.085259 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.085239    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:17.086954 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.086938    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:17.087021 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.086973    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:17.087021 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.086984    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:17.088123 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.088110    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:17.088250 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.088239    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.088293 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.088265    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:17.088833 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.088818    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:17.088922 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.088844    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:17.088922 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.088854    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:17.089391 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.089373    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:17.089495 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.089401    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:17.089495 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.089412    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:17.089956 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.089941    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.090019 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.089973    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:18:17.090613 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.090593    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:18:17.090705 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.090623    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:18:17.090705 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.090636    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:18:17.117965 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:17.117943    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-128.ec2.internal\" not found" node="ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.122086 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:17.122070    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-128.ec2.internal\" not found" node="ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.123108 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.123093    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8047473ee784ce8f8db6eeb4bb41629-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal\" (UID: \"a8047473ee784ce8f8db6eeb4bb41629\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.123159 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.123115    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea53f2068cb53a80ffb0922aa7c95e21-config\") pod \"kube-apiserver-proxy-ip-10-0-143-128.ec2.internal\" (UID: \"ea53f2068cb53a80ffb0922aa7c95e21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.123159 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.123139    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8047473ee784ce8f8db6eeb4bb41629-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal\" (UID: \"a8047473ee784ce8f8db6eeb4bb41629\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.129193 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:17.129179    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-128.ec2.internal\" not found"
Sep 05 02:18:17.224077 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.224052    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8047473ee784ce8f8db6eeb4bb41629-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal\" (UID: \"a8047473ee784ce8f8db6eeb4bb41629\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.224207 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.224079    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea53f2068cb53a80ffb0922aa7c95e21-config\") pod \"kube-apiserver-proxy-ip-10-0-143-128.ec2.internal\" (UID: \"ea53f2068cb53a80ffb0922aa7c95e21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.224207 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.224096    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8047473ee784ce8f8db6eeb4bb41629-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal\" (UID: \"a8047473ee784ce8f8db6eeb4bb41629\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.224207 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.224120    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8047473ee784ce8f8db6eeb4bb41629-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal\" (UID: \"a8047473ee784ce8f8db6eeb4bb41629\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.224207 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.224144    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8047473ee784ce8f8db6eeb4bb41629-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal\" (UID: \"a8047473ee784ce8f8db6eeb4bb41629\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.224207 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.224162    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea53f2068cb53a80ffb0922aa7c95e21-config\") pod \"kube-apiserver-proxy-ip-10-0-143-128.ec2.internal\" (UID: \"ea53f2068cb53a80ffb0922aa7c95e21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.229458 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:17.229431    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-128.ec2.internal\" not found"
Sep 05 02:18:17.330350 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:17.330323    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-128.ec2.internal\" not found"
Sep 05 02:18:17.426601 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.426560    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.426697 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.426624    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:17.430404 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:17.430370    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-128.ec2.internal\" not found"
Sep 05 02:18:17.530950 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:17.530873    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-128.ec2.internal\" not found"
Sep 05 02:18:17.631490 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:17.631460    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-128.ec2.internal\" not found"
Sep 05 02:18:17.697642 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.697617    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:17.731566 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:17.731529    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-128.ec2.internal\" not found"
Sep 05 02:18:17.755040 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.755014    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 05 02:18:17.755183 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.755158    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 02:18:17.755261 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.755203    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 02:18:17.755261 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.755211    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 02:18:17.819236 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.819147    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-04 02:13:17 +0000 UTC" deadline="2028-05-17 11:22:01.87196281 +0000 UTC"
Sep 05 02:18:17.819236 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:17.819195    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14889h3m44.052771224s"
Sep 05 02:18:17.832043 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:17.832012    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-128.ec2.internal\" not found"
Sep 05 02:18:17.932904 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:17.932853    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-128.ec2.internal\" not found"
Sep 05 02:18:18.033134 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:18.033104    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-128.ec2.internal\" not found"
Sep 05 02:18:18.072996 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:18.072927    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8047473ee784ce8f8db6eeb4bb41629.slice/crio-0b61f37299c5d6b880594e3f56d8a6bc4e65b2a6df494155ab59dacec63bee44 WatchSource:0}: Error finding container 0b61f37299c5d6b880594e3f56d8a6bc4e65b2a6df494155ab59dacec63bee44: Status 404 returned error can't find the container with id 0b61f37299c5d6b880594e3f56d8a6bc4e65b2a6df494155ab59dacec63bee44
Sep 05 02:18:18.073134 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:18.073111    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea53f2068cb53a80ffb0922aa7c95e21.slice/crio-da2ec070101a19067455fb35b96987f47c27b227dab5b7f414a6605743268663 WatchSource:0}: Error finding container da2ec070101a19067455fb35b96987f47c27b227dab5b7f414a6605743268663: Status 404 returned error can't find the container with id da2ec070101a19067455fb35b96987f47c27b227dab5b7f414a6605743268663
Sep 05 02:18:18.079827 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.079807    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:18:18.132313 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.132291    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:18.221956 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.221910    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:18.235837 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.235812    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:18:18.237545 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.237180    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-128.ec2.internal"
Sep 05 02:18:18.246203 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.246184    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:18:18.670622 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.670590    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:18.800083 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.800051    2557 apiserver.go:52] "Watching apiserver"
Sep 05 02:18:18.807830 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.807791    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 05 02:18:18.809072 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.809037    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-j9fmm","openshift-multus/multus-gq27j","openshift-network-operator/iptables-alerter-6c7pg","openshift-ovn-kubernetes/ovnkube-node-z4kmp","kube-system/global-pull-secret-syncer-cqpw2","kube-system/konnectivity-agent-dvtdd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj","openshift-cluster-node-tuning-operator/tuned-925z5","openshift-dns/node-resolver-g8xt5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal","openshift-multus/network-metrics-daemon-2c797","openshift-network-diagnostics/network-check-target-2tx7l","kube-system/kube-apiserver-proxy-ip-10-0-143-128.ec2.internal","openshift-image-registry/node-ca-lpd6d"]
Sep 05 02:18:18.810899 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.810624    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.812306 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.812176    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:18.812306 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:18.812258    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:18.813427 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.813400    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zqlfn\""
Sep 05 02:18:18.814094 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.813643    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 05 02:18:18.814094 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.813789    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 05 02:18:18.814094 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.813973    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6c7pg"
Sep 05 02:18:18.814326 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.814107    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 05 02:18:18.814933 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.814916    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.816334 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.816313    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:18.816438 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:18.816414    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:18.816923 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.816695    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:18:18.816923 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.816831    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6wrt7\""
Sep 05 02:18:18.817087 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.817071    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 05 02:18:18.817545 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.817374    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 05 02:18:18.817654 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.817614    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dvtdd"
Sep 05 02:18:18.818327 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.818102    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 05 02:18:18.818429 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.818365    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 05 02:18:18.818813 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.818611    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7ndj9\""
Sep 05 02:18:18.818813 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.818667    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 05 02:18:18.819624 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.819053    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g8xt5"
Sep 05 02:18:18.819624 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.819428    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 05 02:18:18.819624 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.819429    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 05 02:18:18.819958 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.819840    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 05 02:18:18.819958 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.819874    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m4tnv\""
Sep 05 02:18:18.819958 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.819892    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 05 02:18:18.822384 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.820176    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 05 02:18:18.822384 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.820353    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 05 02:18:18.822384 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.820480    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lpd6d"
Sep 05 02:18:18.822384 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.821052    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 05 02:18:18.822384 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.821755    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.822384 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.821884    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 05 02:18:18.825352 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.824873    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p6hvn\""
Sep 05 02:18:18.825352 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.824909    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 05 02:18:18.825352 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.825163    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 05 02:18:18.825763 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.825378    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 05 02:18:18.825763 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.825465    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 05 02:18:18.825763 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.824915    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 05 02:18:18.825763 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.825751    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 05 02:18:18.828839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.825470    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-46kfj\""
Sep 05 02:18:18.828839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.826176    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gmbh2\""
Sep 05 02:18:18.828839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.826247    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:18.828839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.826548    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.828839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.826657    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 05 02:18:18.828839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.827398    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:18.828839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.828226    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:18.828839 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:18.828299    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:18.828839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.828588    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:18:18.829327 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.828984    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 05 02:18:18.830002 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.829952    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 05 02:18:18.830082 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.830065    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qbb6b\""
Sep 05 02:18:18.830374 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.830345    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 05 02:18:18.831158 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.831139    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tmr87\""
Sep 05 02:18:18.834150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834126    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-var-lib-cni-bin\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.834274 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834165    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-var-lib-cni-multus\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.834274 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834222    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43bb8d7d-c160-4ad9-85d5-6b07e88dfd31-konnectivity-ca\") pod \"konnectivity-agent-dvtdd\" (UID: \"43bb8d7d-c160-4ad9-85d5-6b07e88dfd31\") " pod="kube-system/konnectivity-agent-dvtdd"
Sep 05 02:18:18.834274 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834247    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs\") pod \"network-metrics-daemon-2c797\" (UID: \"b933b52f-75e9-478b-a95b-5cb77d236467\") " pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:18.834274 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834273    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/655a8af3-cdba-4d83-b644-a2f072ec2176-host-slash\") pod \"iptables-alerter-6c7pg\" (UID: \"655a8af3-cdba-4d83-b644-a2f072ec2176\") " pod="openshift-network-operator/iptables-alerter-6c7pg"
Sep 05 02:18:18.834554 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834298    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmk88\" (UniqueName: \"kubernetes.io/projected/ad420093-4c18-44bf-b72b-ffa0c853997c-kube-api-access-pmk88\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.834554 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834322    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fee79b98-0963-4527-8484-07cfe3dae8d7-env-overrides\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.834554 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834346    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rff\" (UniqueName: \"kubernetes.io/projected/2836c4ea-8e45-4bd3-9c96-ace613f40ed2-kube-api-access-h2rff\") pod \"node-ca-lpd6d\" (UID: \"2836c4ea-8e45-4bd3-9c96-ace613f40ed2\") " pod="openshift-image-registry/node-ca-lpd6d"
Sep 05 02:18:18.834554 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834371    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/655a8af3-cdba-4d83-b644-a2f072ec2176-iptables-alerter-script\") pod \"iptables-alerter-6c7pg\" (UID: \"655a8af3-cdba-4d83-b644-a2f072ec2176\") " pod="openshift-network-operator/iptables-alerter-6c7pg"
Sep 05 02:18:18.834554 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834395    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-kubelet-config\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:18.834554 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834421    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b-hosts-file\") pod \"node-resolver-g8xt5\" (UID: \"c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b\") " pod="openshift-dns/node-resolver-g8xt5"
Sep 05 02:18:18.834554 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834444    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-run-k8s-cni-cncf-io\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.834554 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834491    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-run-netns\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.834554 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834516    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-multus-conf-dir\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.834554 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834542    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-cni-netd\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834568    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-hostroot\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834592    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834620    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-etc-openvswitch\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834641    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-log-socket\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834664    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fee79b98-0963-4527-8484-07cfe3dae8d7-ovn-node-metrics-cert\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834691    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-multus-socket-dir-parent\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834716    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-registration-dir\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834741    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-run-ovn\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834766    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-multus-cni-dir\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834794    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-cnibin\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834820    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-run-openvswitch\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834849    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-var-lib-kubelet\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834873    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834897    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-systemd-units\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834923    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-run-netns\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.835069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834950    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-node-log\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.834988    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rpn4\" (UniqueName: \"kubernetes.io/projected/b933b52f-75e9-478b-a95b-5cb77d236467-kube-api-access-9rpn4\") pod \"network-metrics-daemon-2c797\" (UID: \"b933b52f-75e9-478b-a95b-5cb77d236467\") " pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835018    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad420093-4c18-44bf-b72b-ffa0c853997c-multus-daemon-config\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835046    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-sys-fs\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835130    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-cni-bin\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835158    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2836c4ea-8e45-4bd3-9c96-ace613f40ed2-host\") pod \"node-ca-lpd6d\" (UID: \"2836c4ea-8e45-4bd3-9c96-ace613f40ed2\") " pod="openshift-image-registry/node-ca-lpd6d"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835218    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b-tmp-dir\") pod \"node-resolver-g8xt5\" (UID: \"c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b\") " pod="openshift-dns/node-resolver-g8xt5"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835261    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-socket-dir\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835291    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-device-dir\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835324    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835349    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43bb8d7d-c160-4ad9-85d5-6b07e88dfd31-agent-certs\") pod \"konnectivity-agent-dvtdd\" (UID: \"43bb8d7d-c160-4ad9-85d5-6b07e88dfd31\") " pod="kube-system/konnectivity-agent-dvtdd"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835375    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rscfw\" (UniqueName: \"kubernetes.io/projected/a85eb80b-fc49-442c-96a8-9d4125c8044a-kube-api-access-rscfw\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835401    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-kubelet\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835439    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-var-lib-openvswitch\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835464    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46r85\" (UniqueName: \"kubernetes.io/projected/fee79b98-0963-4527-8484-07cfe3dae8d7-kube-api-access-46r85\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835490    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:18.835895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835544    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fee79b98-0963-4527-8484-07cfe3dae8d7-ovnkube-config\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.836698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835572    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2836c4ea-8e45-4bd3-9c96-ace613f40ed2-serviceca\") pod \"node-ca-lpd6d\" (UID: \"2836c4ea-8e45-4bd3-9c96-ace613f40ed2\") " pod="openshift-image-registry/node-ca-lpd6d"
Sep 05 02:18:18.836698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835596    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad420093-4c18-44bf-b72b-ffa0c853997c-cni-binary-copy\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.836698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835620    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-etc-kubernetes\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.836698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835701    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-slash\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.836698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835728    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-dbus\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:18.836698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835756    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-system-cni-dir\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.836698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835784    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.836698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835811    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwxbt\" (UniqueName: \"kubernetes.io/projected/c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b-kube-api-access-mwxbt\") pod \"node-resolver-g8xt5\" (UID: \"c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b\") " pod="openshift-dns/node-resolver-g8xt5"
Sep 05 02:18:18.836698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835837    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fee79b98-0963-4527-8484-07cfe3dae8d7-ovnkube-script-lib\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.836698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835879    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7pvw\" (UniqueName: \"kubernetes.io/projected/655a8af3-cdba-4d83-b644-a2f072ec2176-kube-api-access-w7pvw\") pod \"iptables-alerter-6c7pg\" (UID: \"655a8af3-cdba-4d83-b644-a2f072ec2176\") " pod="openshift-network-operator/iptables-alerter-6c7pg"
Sep 05 02:18:18.836698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835922    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-run-multus-certs\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.836698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.835981    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-run-systemd\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.840569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.838744    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:18:18.868179 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.868150    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pt9w6"
Sep 05 02:18:18.874607 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.874588    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pt9w6"
Sep 05 02:18:18.923604 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.923545    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936428    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs\") pod \"network-metrics-daemon-2c797\" (UID: \"b933b52f-75e9-478b-a95b-5cb77d236467\") " pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936482    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/655a8af3-cdba-4d83-b644-a2f072ec2176-host-slash\") pod \"iptables-alerter-6c7pg\" (UID: \"655a8af3-cdba-4d83-b644-a2f072ec2176\") " pod="openshift-network-operator/iptables-alerter-6c7pg"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936509    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmk88\" (UniqueName: \"kubernetes.io/projected/ad420093-4c18-44bf-b72b-ffa0c853997c-kube-api-access-pmk88\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936534    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fee79b98-0963-4527-8484-07cfe3dae8d7-env-overrides\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936560    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rff\" (UniqueName: \"kubernetes.io/projected/2836c4ea-8e45-4bd3-9c96-ace613f40ed2-kube-api-access-h2rff\") pod \"node-ca-lpd6d\" (UID: \"2836c4ea-8e45-4bd3-9c96-ace613f40ed2\") " pod="openshift-image-registry/node-ca-lpd6d"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:18.936578    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936587    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/655a8af3-cdba-4d83-b644-a2f072ec2176-iptables-alerter-script\") pod \"iptables-alerter-6c7pg\" (UID: \"655a8af3-cdba-4d83-b644-a2f072ec2176\") " pod="openshift-network-operator/iptables-alerter-6c7pg"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936614    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-kubelet-config\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936639    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b-hosts-file\") pod \"node-resolver-g8xt5\" (UID: \"c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b\") " pod="openshift-dns/node-resolver-g8xt5"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:18.936672    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs podName:b933b52f-75e9-478b-a95b-5cb77d236467 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:19.436635314 +0000 UTC m=+2.991791625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs") pod "network-metrics-daemon-2c797" (UID: "b933b52f-75e9-478b-a95b-5cb77d236467") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936708    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b-hosts-file\") pod \"node-resolver-g8xt5\" (UID: \"c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b\") " pod="openshift-dns/node-resolver-g8xt5"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936706    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-run-k8s-cni-cncf-io\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936743    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-run-netns\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936751    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-run-k8s-cni-cncf-io\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936767    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-multus-conf-dir\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936795    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-cni-netd\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.941685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936883    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-multus-conf-dir\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936914    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-hostroot\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936942    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.936999    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.937200    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-cni-netd\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.937256    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/655a8af3-cdba-4d83-b644-a2f072ec2176-host-slash\") pod \"iptables-alerter-6c7pg\" (UID: \"655a8af3-cdba-4d83-b644-a2f072ec2176\") " pod="openshift-network-operator/iptables-alerter-6c7pg"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.937468    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-etc-openvswitch\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.937588    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/655a8af3-cdba-4d83-b644-a2f072ec2176-iptables-alerter-script\") pod \"iptables-alerter-6c7pg\" (UID: \"655a8af3-cdba-4d83-b644-a2f072ec2176\") " pod="openshift-network-operator/iptables-alerter-6c7pg"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.937678    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-kubelet-config\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.937720    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-run-netns\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.937758    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-hostroot\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.937794    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-log-socket\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.937841    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fee79b98-0963-4527-8484-07cfe3dae8d7-ovn-node-metrics-cert\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.937869    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-multus-socket-dir-parent\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.937918    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-registration-dir\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.937945    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-run-ovn\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.937965    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fee79b98-0963-4527-8484-07cfe3dae8d7-env-overrides\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.942651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.937975    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cztf6\" (UniqueName: \"kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6\") pod \"network-check-target-2tx7l\" (UID: \"1761ac29-4d31-4d57-9355-5cffbdc2ef2c\") " pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938006    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-multus-cni-dir\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938016    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-etc-openvswitch\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938057    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-cnibin\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938069    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-log-socket\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938083    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-run-openvswitch\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938109    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-sysconfig\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938122    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-multus-cni-dir\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938141    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-tuned\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938181    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-multus-socket-dir-parent\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938256    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-var-lib-kubelet\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938288    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-var-lib-kubelet\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938303    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-run-ovn\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938337    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-cnibin\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938351    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938376    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-run-openvswitch\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938411    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-systemd-units\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.943501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938451    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-registration-dir\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938507    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-lib-modules\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938548    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a78d01f0-d41b-40ad-8d73-ddc3df30e814-system-cni-dir\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938601    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-run-netns\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938630    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-node-log\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938659    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a78d01f0-d41b-40ad-8d73-ddc3df30e814-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938689    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rpn4\" (UniqueName: \"kubernetes.io/projected/b933b52f-75e9-478b-a95b-5cb77d236467-kube-api-access-9rpn4\") pod \"network-metrics-daemon-2c797\" (UID: \"b933b52f-75e9-478b-a95b-5cb77d236467\") " pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938756    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad420093-4c18-44bf-b72b-ffa0c853997c-multus-daemon-config\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938784    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-sys-fs\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938810    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-cni-bin\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938838    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-modprobe-d\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938840    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938866    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-sys\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938890    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a0adfb7-0d26-4292-8790-81e38ba82c1a-tmp\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938914    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a78d01f0-d41b-40ad-8d73-ddc3df30e814-cnibin\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.938943    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2836c4ea-8e45-4bd3-9c96-ace613f40ed2-host\") pod \"node-ca-lpd6d\" (UID: \"2836c4ea-8e45-4bd3-9c96-ace613f40ed2\") " pod="openshift-image-registry/node-ca-lpd6d"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939028    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b-tmp-dir\") pod \"node-resolver-g8xt5\" (UID: \"c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b\") " pod="openshift-dns/node-resolver-g8xt5"
Sep 05 02:18:18.944569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939072    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-socket-dir\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939100    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-device-dir\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939131    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939159    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43bb8d7d-c160-4ad9-85d5-6b07e88dfd31-agent-certs\") pod \"konnectivity-agent-dvtdd\" (UID: \"43bb8d7d-c160-4ad9-85d5-6b07e88dfd31\") " pod="kube-system/konnectivity-agent-dvtdd"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939187    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-sysctl-conf\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939223    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rscfw\" (UniqueName: \"kubernetes.io/projected/a85eb80b-fc49-442c-96a8-9d4125c8044a-kube-api-access-rscfw\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939256    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-kubelet\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939282    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-var-lib-openvswitch\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939312    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46r85\" (UniqueName: \"kubernetes.io/projected/fee79b98-0963-4527-8484-07cfe3dae8d7-kube-api-access-46r85\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939339    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-systemd\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939376    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-run\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939400    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-host\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939428    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znq8d\" (UniqueName: \"kubernetes.io/projected/a78d01f0-d41b-40ad-8d73-ddc3df30e814-kube-api-access-znq8d\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939464    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939537    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2836c4ea-8e45-4bd3-9c96-ace613f40ed2-host\") pod \"node-ca-lpd6d\" (UID: \"2836c4ea-8e45-4bd3-9c96-ace613f40ed2\") " pod="openshift-image-registry/node-ca-lpd6d"
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:18.939561    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:18.945414 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:18.939614    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret podName:39f296d4-d2f6-46c7-ad0d-b520ecb0138e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:19.439596814 +0000 UTC m=+2.994753104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret") pod "global-pull-secret-syncer-cqpw2" (UID: "39f296d4-d2f6-46c7-ad0d-b520ecb0138e") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939769    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-systemd-units\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939817    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-run-netns\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939855    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-node-log\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.939890    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b-tmp-dir\") pod \"node-resolver-g8xt5\" (UID: \"c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b\") " pod="openshift-dns/node-resolver-g8xt5"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940017    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-socket-dir\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940075    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-device-dir\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940124    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940380    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad420093-4c18-44bf-b72b-ffa0c853997c-multus-daemon-config\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940497    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a85eb80b-fc49-442c-96a8-9d4125c8044a-sys-fs\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940505    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-cni-bin\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940572    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fee79b98-0963-4527-8484-07cfe3dae8d7-ovnkube-config\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940645    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a78d01f0-d41b-40ad-8d73-ddc3df30e814-cni-binary-copy\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940681    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2836c4ea-8e45-4bd3-9c96-ace613f40ed2-serviceca\") pod \"node-ca-lpd6d\" (UID: \"2836c4ea-8e45-4bd3-9c96-ace613f40ed2\") " pod="openshift-image-registry/node-ca-lpd6d"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940724    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad420093-4c18-44bf-b72b-ffa0c853997c-cni-binary-copy\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940772    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-etc-kubernetes\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940800    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-slash\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.946312 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940846    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-dbus\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940881    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-var-lib-openvswitch\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940882    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-system-cni-dir\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940850    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-slash\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940960    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940978    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-system-cni-dir\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.940991    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a78d01f0-d41b-40ad-8d73-ddc3df30e814-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941023    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwxbt\" (UniqueName: \"kubernetes.io/projected/c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b-kube-api-access-mwxbt\") pod \"node-resolver-g8xt5\" (UID: \"c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b\") " pod="openshift-dns/node-resolver-g8xt5"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941050    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fee79b98-0963-4527-8484-07cfe3dae8d7-ovnkube-script-lib\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941077    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7pvw\" (UniqueName: \"kubernetes.io/projected/655a8af3-cdba-4d83-b644-a2f072ec2176-kube-api-access-w7pvw\") pod \"iptables-alerter-6c7pg\" (UID: \"655a8af3-cdba-4d83-b644-a2f072ec2176\") " pod="openshift-network-operator/iptables-alerter-6c7pg"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941106    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-run-multus-certs\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941132    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-run-systemd\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941158    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-var-lib-kubelet\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941197    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-var-lib-cni-bin\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941226    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-var-lib-cni-multus\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941298    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43bb8d7d-c160-4ad9-85d5-6b07e88dfd31-konnectivity-ca\") pod \"konnectivity-agent-dvtdd\" (UID: \"43bb8d7d-c160-4ad9-85d5-6b07e88dfd31\") " pod="kube-system/konnectivity-agent-dvtdd"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941325    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-kubernetes\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.947214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941351    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-sysctl-d\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941379    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqrdq\" (UniqueName: \"kubernetes.io/projected/9a0adfb7-0d26-4292-8790-81e38ba82c1a-kube-api-access-sqrdq\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941407    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a78d01f0-d41b-40ad-8d73-ddc3df30e814-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941458    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad420093-4c18-44bf-b72b-ffa0c853997c-cni-binary-copy\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941486    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-var-lib-cni-multus\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941542    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-etc-kubernetes\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941594    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-dbus\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941812    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2836c4ea-8e45-4bd3-9c96-ace613f40ed2-serviceca\") pod \"node-ca-lpd6d\" (UID: \"2836c4ea-8e45-4bd3-9c96-ace613f40ed2\") " pod="openshift-image-registry/node-ca-lpd6d"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941913    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-run-systemd\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.941968    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.942041    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fee79b98-0963-4527-8484-07cfe3dae8d7-host-kubelet\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.942052    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fee79b98-0963-4527-8484-07cfe3dae8d7-ovnkube-config\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.942454    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-run-multus-certs\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.942549    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fee79b98-0963-4527-8484-07cfe3dae8d7-ovnkube-script-lib\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.942595    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad420093-4c18-44bf-b72b-ffa0c853997c-host-var-lib-cni-bin\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.942633    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43bb8d7d-c160-4ad9-85d5-6b07e88dfd31-konnectivity-ca\") pod \"konnectivity-agent-dvtdd\" (UID: \"43bb8d7d-c160-4ad9-85d5-6b07e88dfd31\") " pod="kube-system/konnectivity-agent-dvtdd"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.946132    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fee79b98-0963-4527-8484-07cfe3dae8d7-ovn-node-metrics-cert\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.948076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.946815    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43bb8d7d-c160-4ad9-85d5-6b07e88dfd31-agent-certs\") pod \"konnectivity-agent-dvtdd\" (UID: \"43bb8d7d-c160-4ad9-85d5-6b07e88dfd31\") " pod="kube-system/konnectivity-agent-dvtdd"
Sep 05 02:18:18.950696 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.950670    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmk88\" (UniqueName: \"kubernetes.io/projected/ad420093-4c18-44bf-b72b-ffa0c853997c-kube-api-access-pmk88\") pod \"multus-gq27j\" (UID: \"ad420093-4c18-44bf-b72b-ffa0c853997c\") " pod="openshift-multus/multus-gq27j"
Sep 05 02:18:18.951807 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.951782    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rff\" (UniqueName: \"kubernetes.io/projected/2836c4ea-8e45-4bd3-9c96-ace613f40ed2-kube-api-access-h2rff\") pod \"node-ca-lpd6d\" (UID: \"2836c4ea-8e45-4bd3-9c96-ace613f40ed2\") " pod="openshift-image-registry/node-ca-lpd6d"
Sep 05 02:18:18.954888 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.954865    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7pvw\" (UniqueName: \"kubernetes.io/projected/655a8af3-cdba-4d83-b644-a2f072ec2176-kube-api-access-w7pvw\") pod \"iptables-alerter-6c7pg\" (UID: \"655a8af3-cdba-4d83-b644-a2f072ec2176\") " pod="openshift-network-operator/iptables-alerter-6c7pg"
Sep 05 02:18:18.955194 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.955167    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwxbt\" (UniqueName: \"kubernetes.io/projected/c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b-kube-api-access-mwxbt\") pod \"node-resolver-g8xt5\" (UID: \"c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b\") " pod="openshift-dns/node-resolver-g8xt5"
Sep 05 02:18:18.957200 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.957163    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rscfw\" (UniqueName: \"kubernetes.io/projected/a85eb80b-fc49-442c-96a8-9d4125c8044a-kube-api-access-rscfw\") pod \"aws-ebs-csi-driver-node-4qnpj\" (UID: \"a85eb80b-fc49-442c-96a8-9d4125c8044a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:18.957299 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.957283    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rpn4\" (UniqueName: \"kubernetes.io/projected/b933b52f-75e9-478b-a95b-5cb77d236467-kube-api-access-9rpn4\") pod \"network-metrics-daemon-2c797\" (UID: \"b933b52f-75e9-478b-a95b-5cb77d236467\") " pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:18.957638 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.957579    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46r85\" (UniqueName: \"kubernetes.io/projected/fee79b98-0963-4527-8484-07cfe3dae8d7-kube-api-access-46r85\") pod \"ovnkube-node-z4kmp\" (UID: \"fee79b98-0963-4527-8484-07cfe3dae8d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:18.990206 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.990146    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-128.ec2.internal" event={"ID":"ea53f2068cb53a80ffb0922aa7c95e21","Type":"ContainerStarted","Data":"da2ec070101a19067455fb35b96987f47c27b227dab5b7f414a6605743268663"}
Sep 05 02:18:18.991810 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:18.991769    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal" event={"ID":"a8047473ee784ce8f8db6eeb4bb41629","Type":"ContainerStarted","Data":"0b61f37299c5d6b880594e3f56d8a6bc4e65b2a6df494155ab59dacec63bee44"}
Sep 05 02:18:19.042316 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042280    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a78d01f0-d41b-40ad-8d73-ddc3df30e814-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.042472 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042328    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-var-lib-kubelet\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.042472 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042359    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-kubernetes\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.042472 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042386    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-sysctl-d\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.042472 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042411    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqrdq\" (UniqueName: \"kubernetes.io/projected/9a0adfb7-0d26-4292-8790-81e38ba82c1a-kube-api-access-sqrdq\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.042472 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042438    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a78d01f0-d41b-40ad-8d73-ddc3df30e814-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.042728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042474    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-var-lib-kubelet\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.042728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042541    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-kubernetes\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.042728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042578    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztf6\" (UniqueName: \"kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6\") pod \"network-check-target-2tx7l\" (UID: \"1761ac29-4d31-4d57-9355-5cffbdc2ef2c\") " pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:19.042728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042605    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-sysconfig\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.042728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042632    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-tuned\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.042728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042661    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-lib-modules\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.042728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042672    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a78d01f0-d41b-40ad-8d73-ddc3df30e814-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.042728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042686    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a78d01f0-d41b-40ad-8d73-ddc3df30e814-system-cni-dir\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.042728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042712    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a78d01f0-d41b-40ad-8d73-ddc3df30e814-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042729    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-sysconfig\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042738    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-modprobe-d\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042793    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-sys\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042812    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-modprobe-d\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042812    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-sysctl-d\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042818    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a0adfb7-0d26-4292-8790-81e38ba82c1a-tmp\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042856    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a78d01f0-d41b-40ad-8d73-ddc3df30e814-cnibin\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042867    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-sys\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042888    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-sysctl-conf\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042917    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-systemd\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042940    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-run\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042966    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-host\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.042991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znq8d\" (UniqueName: \"kubernetes.io/projected/a78d01f0-d41b-40ad-8d73-ddc3df30e814-kube-api-access-znq8d\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.043030    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a78d01f0-d41b-40ad-8d73-ddc3df30e814-cni-binary-copy\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.043160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.043125    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-lib-modules\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.043236    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-systemd\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.043285    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a78d01f0-d41b-40ad-8d73-ddc3df30e814-cnibin\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.043917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.043290    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-host\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.043349    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-run\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.043388    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-sysctl-conf\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.043917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.043441    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a78d01f0-d41b-40ad-8d73-ddc3df30e814-system-cni-dir\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.043917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.043730    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a78d01f0-d41b-40ad-8d73-ddc3df30e814-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.043917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.043783    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a78d01f0-d41b-40ad-8d73-ddc3df30e814-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.043917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.043838    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a78d01f0-d41b-40ad-8d73-ddc3df30e814-cni-binary-copy\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.045713 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.045676    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a0adfb7-0d26-4292-8790-81e38ba82c1a-tmp\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.045819 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.045742    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a0adfb7-0d26-4292-8790-81e38ba82c1a-etc-tuned\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.048955 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:19.048929    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:19.048955 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:19.048956    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:19.049131 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:19.048972    2557 projected.go:196] Error preparing data for projected volume kube-api-access-cztf6 for pod openshift-network-diagnostics/network-check-target-2tx7l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:19.049131 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:19.049055    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6 podName:1761ac29-4d31-4d57-9355-5cffbdc2ef2c nodeName:}" failed. No retries permitted until 2026-09-05 02:18:19.549037165 +0000 UTC m=+3.104193478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cztf6" (UniqueName: "kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6") pod "network-check-target-2tx7l" (UID: "1761ac29-4d31-4d57-9355-5cffbdc2ef2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:19.053338 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.053309    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znq8d\" (UniqueName: \"kubernetes.io/projected/a78d01f0-d41b-40ad-8d73-ddc3df30e814-kube-api-access-znq8d\") pod \"multus-additional-cni-plugins-j9fmm\" (UID: \"a78d01f0-d41b-40ad-8d73-ddc3df30e814\") " pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.054474 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.054406    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqrdq\" (UniqueName: \"kubernetes.io/projected/9a0adfb7-0d26-4292-8790-81e38ba82c1a-kube-api-access-sqrdq\") pod \"tuned-925z5\" (UID: \"9a0adfb7-0d26-4292-8790-81e38ba82c1a\") " pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.125497 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.125463    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj"
Sep 05 02:18:19.136579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.136555    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6c7pg"
Sep 05 02:18:19.146864 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.146836    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:19.153798 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.153767    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dvtdd"
Sep 05 02:18:19.162129 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.162105    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g8xt5"
Sep 05 02:18:19.172778 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.172756    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lpd6d"
Sep 05 02:18:19.179671 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.179647    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gq27j"
Sep 05 02:18:19.189654 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.189635    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-925z5"
Sep 05 02:18:19.195296 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.195270    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j9fmm"
Sep 05 02:18:19.446059 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.445988    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:19.446059 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.446044    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs\") pod \"network-metrics-daemon-2c797\" (UID: \"b933b52f-75e9-478b-a95b-5cb77d236467\") " pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:19.446235 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:19.446175    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:19.446235 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:19.446178    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:19.446342 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:19.446247    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs podName:b933b52f-75e9-478b-a95b-5cb77d236467 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:20.446226825 +0000 UTC m=+4.001383117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs") pod "network-metrics-daemon-2c797" (UID: "b933b52f-75e9-478b-a95b-5cb77d236467") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:19.446342 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:19.446266    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret podName:39f296d4-d2f6-46c7-ad0d-b520ecb0138e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:20.446256544 +0000 UTC m=+4.001412837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret") pod "global-pull-secret-syncer-cqpw2" (UID: "39f296d4-d2f6-46c7-ad0d-b520ecb0138e") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:19.647456 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.647416    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztf6\" (UniqueName: \"kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6\") pod \"network-check-target-2tx7l\" (UID: \"1761ac29-4d31-4d57-9355-5cffbdc2ef2c\") " pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:19.647650 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:19.647614    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:19.647650 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:19.647636    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:19.647650 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:19.647646    2557 projected.go:196] Error preparing data for projected volume kube-api-access-cztf6 for pod openshift-network-diagnostics/network-check-target-2tx7l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:19.647794 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:19.647703    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6 podName:1761ac29-4d31-4d57-9355-5cffbdc2ef2c nodeName:}" failed. No retries permitted until 2026-09-05 02:18:20.64768828 +0000 UTC m=+4.202844574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cztf6" (UniqueName: "kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6") pod "network-check-target-2tx7l" (UID: "1761ac29-4d31-4d57-9355-5cffbdc2ef2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:19.878392 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.878295    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:13:19 +0000 UTC" deadline="2028-03-12 05:32:44.256389282 +0000 UTC"
Sep 05 02:18:19.878392 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.878328    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13299h14m24.37806464s"
Sep 05 02:18:19.985393 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.985361    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:19.985393 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:19.985387    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:19.985644 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:19.985530    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:19.985734 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:19.985703    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:20.107248 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:20.107017    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10e20bb_ff5e_4f8f_82c1_7f12be57bd1b.slice/crio-5446a0486d267901e1a7c39e0ed80c917d016f2d210d13a426912203717888e1 WatchSource:0}: Error finding container 5446a0486d267901e1a7c39e0ed80c917d016f2d210d13a426912203717888e1: Status 404 returned error can't find the container with id 5446a0486d267901e1a7c39e0ed80c917d016f2d210d13a426912203717888e1
Sep 05 02:18:20.108370 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:20.108342    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43bb8d7d_c160_4ad9_85d5_6b07e88dfd31.slice/crio-ff3e4b4b5fc331467c3281b56f65cda3af1b82e6b9e05378205c2270472c38db WatchSource:0}: Error finding container ff3e4b4b5fc331467c3281b56f65cda3af1b82e6b9e05378205c2270472c38db: Status 404 returned error can't find the container with id ff3e4b4b5fc331467c3281b56f65cda3af1b82e6b9e05378205c2270472c38db
Sep 05 02:18:20.109782 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:20.109753    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78d01f0_d41b_40ad_8d73_ddc3df30e814.slice/crio-ca54169e46db857aea3de1d4a9d3fe580f55a75c0472a9e9056bea14e0ba1b73 WatchSource:0}: Error finding container ca54169e46db857aea3de1d4a9d3fe580f55a75c0472a9e9056bea14e0ba1b73: Status 404 returned error can't find the container with id ca54169e46db857aea3de1d4a9d3fe580f55a75c0472a9e9056bea14e0ba1b73
Sep 05 02:18:20.111500 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:20.111393    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0adfb7_0d26_4292_8790_81e38ba82c1a.slice/crio-2edf28cc148cf27d357db318134abe38485ff21523ae5bbc36ea679675173d12 WatchSource:0}: Error finding container 2edf28cc148cf27d357db318134abe38485ff21523ae5bbc36ea679675173d12: Status 404 returned error can't find the container with id 2edf28cc148cf27d357db318134abe38485ff21523ae5bbc36ea679675173d12
Sep 05 02:18:20.114881 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:20.114857    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85eb80b_fc49_442c_96a8_9d4125c8044a.slice/crio-19e350c5899924e4eff8be64c1c38da7bace1f9494ec6ec1aadbbc473d7b12e4 WatchSource:0}: Error finding container 19e350c5899924e4eff8be64c1c38da7bace1f9494ec6ec1aadbbc473d7b12e4: Status 404 returned error can't find the container with id 19e350c5899924e4eff8be64c1c38da7bace1f9494ec6ec1aadbbc473d7b12e4
Sep 05 02:18:20.115889 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:20.115718    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2836c4ea_8e45_4bd3_9c96_ace613f40ed2.slice/crio-3b337327eb34e7802e6c0b3f3556b5a1b2038698d23580765afec3a7561bb4fc WatchSource:0}: Error finding container 3b337327eb34e7802e6c0b3f3556b5a1b2038698d23580765afec3a7561bb4fc: Status 404 returned error can't find the container with id 3b337327eb34e7802e6c0b3f3556b5a1b2038698d23580765afec3a7561bb4fc
Sep 05 02:18:20.116837 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:20.116788    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee79b98_0963_4527_8484_07cfe3dae8d7.slice/crio-b0014192629e2aa582982053ee3d7ff68227de62c05c348b973a830c149f7b58 WatchSource:0}: Error finding container b0014192629e2aa582982053ee3d7ff68227de62c05c348b973a830c149f7b58: Status 404 returned error can't find the container with id b0014192629e2aa582982053ee3d7ff68227de62c05c348b973a830c149f7b58
Sep 05 02:18:20.118308 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:20.118022    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655a8af3_cdba_4d83_b644_a2f072ec2176.slice/crio-93b12a082036248f27db6e63841a361ea8c221d5cef3f6875221cf93302c7ee7 WatchSource:0}: Error finding container 93b12a082036248f27db6e63841a361ea8c221d5cef3f6875221cf93302c7ee7: Status 404 returned error can't find the container with id 93b12a082036248f27db6e63841a361ea8c221d5cef3f6875221cf93302c7ee7
Sep 05 02:18:20.120417 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:20.120315    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad420093_4c18_44bf_b72b_ffa0c853997c.slice/crio-2c627e8d6a5da498ff886a7be59febe6bf876c0ce337fb35598a6280ea6d96a9 WatchSource:0}: Error finding container 2c627e8d6a5da498ff886a7be59febe6bf876c0ce337fb35598a6280ea6d96a9: Status 404 returned error can't find the container with id 2c627e8d6a5da498ff886a7be59febe6bf876c0ce337fb35598a6280ea6d96a9
Sep 05 02:18:20.160774 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:20.160744    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:18:20.454733 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:20.454696    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:20.454866 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:20.454750    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs\") pod \"network-metrics-daemon-2c797\" (UID: \"b933b52f-75e9-478b-a95b-5cb77d236467\") " pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:20.454922 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:20.454855    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:20.454959 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:20.454930    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret podName:39f296d4-d2f6-46c7-ad0d-b520ecb0138e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:22.454913144 +0000 UTC m=+6.010069433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret") pod "global-pull-secret-syncer-cqpw2" (UID: "39f296d4-d2f6-46c7-ad0d-b520ecb0138e") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:20.455005 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:20.454868    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:20.455036 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:20.455009    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs podName:b933b52f-75e9-478b-a95b-5cb77d236467 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:22.4549965 +0000 UTC m=+6.010152803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs") pod "network-metrics-daemon-2c797" (UID: "b933b52f-75e9-478b-a95b-5cb77d236467") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:20.657092 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:20.657003    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztf6\" (UniqueName: \"kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6\") pod \"network-check-target-2tx7l\" (UID: \"1761ac29-4d31-4d57-9355-5cffbdc2ef2c\") " pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:20.657265 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:20.657219    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:20.657265 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:20.657247    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:20.657265 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:20.657260    2557 projected.go:196] Error preparing data for projected volume kube-api-access-cztf6 for pod openshift-network-diagnostics/network-check-target-2tx7l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:20.657431 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:20.657322    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6 podName:1761ac29-4d31-4d57-9355-5cffbdc2ef2c nodeName:}" failed. No retries permitted until 2026-09-05 02:18:22.657302294 +0000 UTC m=+6.212458599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cztf6" (UniqueName: "kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6") pod "network-check-target-2tx7l" (UID: "1761ac29-4d31-4d57-9355-5cffbdc2ef2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:20.878566 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:20.878466    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:13:19 +0000 UTC" deadline="2028-02-08 11:17:56.770640172 +0000 UTC"
Sep 05 02:18:20.878566 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:20.878530    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12512h59m35.892114549s"
Sep 05 02:18:20.987821 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:20.987789    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:20.987988 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:20.987937    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:21.008733 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.008651    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g8xt5" event={"ID":"c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b","Type":"ContainerStarted","Data":"5446a0486d267901e1a7c39e0ed80c917d016f2d210d13a426912203717888e1"}
Sep 05 02:18:21.015116 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.015078    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6c7pg" event={"ID":"655a8af3-cdba-4d83-b644-a2f072ec2176","Type":"ContainerStarted","Data":"93b12a082036248f27db6e63841a361ea8c221d5cef3f6875221cf93302c7ee7"}
Sep 05 02:18:21.025396 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.025359    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp" event={"ID":"fee79b98-0963-4527-8484-07cfe3dae8d7","Type":"ContainerStarted","Data":"b0014192629e2aa582982053ee3d7ff68227de62c05c348b973a830c149f7b58"}
Sep 05 02:18:21.036203 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.036170    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j9fmm" event={"ID":"a78d01f0-d41b-40ad-8d73-ddc3df30e814","Type":"ContainerStarted","Data":"ca54169e46db857aea3de1d4a9d3fe580f55a75c0472a9e9056bea14e0ba1b73"}
Sep 05 02:18:21.040250 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.040220    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dvtdd" event={"ID":"43bb8d7d-c160-4ad9-85d5-6b07e88dfd31","Type":"ContainerStarted","Data":"ff3e4b4b5fc331467c3281b56f65cda3af1b82e6b9e05378205c2270472c38db"}
Sep 05 02:18:21.048795 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.048765    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-128.ec2.internal" event={"ID":"ea53f2068cb53a80ffb0922aa7c95e21","Type":"ContainerStarted","Data":"dd47624da7cb536187e6651f2212cea2af7c51b289b1b93b3e607894bdf480e2"}
Sep 05 02:18:21.057013 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.056985    2557 generic.go:356] "Generic (PLEG): container finished" podID="a8047473ee784ce8f8db6eeb4bb41629" containerID="defaf96048bf67b1f0622a13fe34e688bcfb3024fd113c659c6ecd8d6caa9d33" exitCode=0
Sep 05 02:18:21.057602 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.057579    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal" event={"ID":"a8047473ee784ce8f8db6eeb4bb41629","Type":"ContainerDied","Data":"defaf96048bf67b1f0622a13fe34e688bcfb3024fd113c659c6ecd8d6caa9d33"}
Sep 05 02:18:21.061039 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.061014    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gq27j" event={"ID":"ad420093-4c18-44bf-b72b-ffa0c853997c","Type":"ContainerStarted","Data":"2c627e8d6a5da498ff886a7be59febe6bf876c0ce337fb35598a6280ea6d96a9"}
Sep 05 02:18:21.071363 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.071336    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lpd6d" event={"ID":"2836c4ea-8e45-4bd3-9c96-ace613f40ed2","Type":"ContainerStarted","Data":"3b337327eb34e7802e6c0b3f3556b5a1b2038698d23580765afec3a7561bb4fc"}
Sep 05 02:18:21.085954 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.085899    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-128.ec2.internal" podStartSLOduration=3.085881583 podStartE2EDuration="3.085881583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:21.068328702 +0000 UTC m=+4.623485016" watchObservedRunningTime="2026-09-05 02:18:21.085881583 +0000 UTC m=+4.641037896"
Sep 05 02:18:21.094340 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.094308    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj" event={"ID":"a85eb80b-fc49-442c-96a8-9d4125c8044a","Type":"ContainerStarted","Data":"19e350c5899924e4eff8be64c1c38da7bace1f9494ec6ec1aadbbc473d7b12e4"}
Sep 05 02:18:21.110120 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.110088    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-925z5" event={"ID":"9a0adfb7-0d26-4292-8790-81e38ba82c1a","Type":"ContainerStarted","Data":"2edf28cc148cf27d357db318134abe38485ff21523ae5bbc36ea679675173d12"}
Sep 05 02:18:21.985134 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.984574    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:21.985134 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:21.984580    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:21.985134 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:21.984723    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:21.985134 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:21.984784    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:22.137731 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:22.137648    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal" event={"ID":"a8047473ee784ce8f8db6eeb4bb41629","Type":"ContainerStarted","Data":"99d6fa79b63db4fa955d66bce3a0e9ce20c805d4035ef4b977766c2e9c3b5f4c"}
Sep 05 02:18:22.475439 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:22.474836    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:22.475439 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:22.474900    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs\") pod \"network-metrics-daemon-2c797\" (UID: \"b933b52f-75e9-478b-a95b-5cb77d236467\") " pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:22.475439 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:22.475020    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:22.475439 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:22.475090    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs podName:b933b52f-75e9-478b-a95b-5cb77d236467 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:26.475067565 +0000 UTC m=+10.030223861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs") pod "network-metrics-daemon-2c797" (UID: "b933b52f-75e9-478b-a95b-5cb77d236467") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:22.475439 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:22.475019    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:22.475439 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:22.475173    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret podName:39f296d4-d2f6-46c7-ad0d-b520ecb0138e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:26.475149795 +0000 UTC m=+10.030306097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret") pod "global-pull-secret-syncer-cqpw2" (UID: "39f296d4-d2f6-46c7-ad0d-b520ecb0138e") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:22.676222 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:22.676123    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztf6\" (UniqueName: \"kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6\") pod \"network-check-target-2tx7l\" (UID: \"1761ac29-4d31-4d57-9355-5cffbdc2ef2c\") " pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:22.676400 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:22.676315    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:22.676400 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:22.676353    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:22.676400 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:22.676369    2557 projected.go:196] Error preparing data for projected volume kube-api-access-cztf6 for pod openshift-network-diagnostics/network-check-target-2tx7l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:22.676624 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:22.676436    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6 podName:1761ac29-4d31-4d57-9355-5cffbdc2ef2c nodeName:}" failed. No retries permitted until 2026-09-05 02:18:26.676415298 +0000 UTC m=+10.231571613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cztf6" (UniqueName: "kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6") pod "network-check-target-2tx7l" (UID: "1761ac29-4d31-4d57-9355-5cffbdc2ef2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:22.984494 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:22.984458    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:22.984704 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:22.984607    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:23.984714 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:23.984678    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:23.985170 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:23.984813    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:23.985170 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:23.984865    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:23.985170 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:23.985001    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:24.984410 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:24.984381    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:24.984606 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:24.984522    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:25.984681 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:25.984648    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:25.985127 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:25.984784    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:25.985127 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:25.984844    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:25.985127 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:25.984955    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:26.511121 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:26.510493    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs\") pod \"network-metrics-daemon-2c797\" (UID: \"b933b52f-75e9-478b-a95b-5cb77d236467\") " pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:26.511121 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:26.510580    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:26.511121 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:26.510699    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:26.511121 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:26.510752    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret podName:39f296d4-d2f6-46c7-ad0d-b520ecb0138e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:34.51073625 +0000 UTC m=+18.065892536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret") pod "global-pull-secret-syncer-cqpw2" (UID: "39f296d4-d2f6-46c7-ad0d-b520ecb0138e") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:26.511121 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:26.510809    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:26.511121 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:26.510832    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs podName:b933b52f-75e9-478b-a95b-5cb77d236467 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:34.510823255 +0000 UTC m=+18.065979542 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs") pod "network-metrics-daemon-2c797" (UID: "b933b52f-75e9-478b-a95b-5cb77d236467") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:26.712690 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:26.712655    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztf6\" (UniqueName: \"kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6\") pod \"network-check-target-2tx7l\" (UID: \"1761ac29-4d31-4d57-9355-5cffbdc2ef2c\") " pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:26.712858 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:26.712832    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:26.712858 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:26.712853    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:26.712960 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:26.712865    2557 projected.go:196] Error preparing data for projected volume kube-api-access-cztf6 for pod openshift-network-diagnostics/network-check-target-2tx7l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:26.712960 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:26.712923    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6 podName:1761ac29-4d31-4d57-9355-5cffbdc2ef2c nodeName:}" failed. No retries permitted until 2026-09-05 02:18:34.712905418 +0000 UTC m=+18.268061719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cztf6" (UniqueName: "kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6") pod "network-check-target-2tx7l" (UID: "1761ac29-4d31-4d57-9355-5cffbdc2ef2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:26.988097 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:26.986889    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:26.988097 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:26.987017    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:27.984819 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:27.984616    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:27.985038 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:27.984817    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:27.985154 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:27.985110    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:27.985387 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:27.985322    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:28.984728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:28.984647    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:28.985157 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:28.984785    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:29.984895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:29.984863    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:29.985354 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:29.984910    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:29.985354 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:29.985026    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:29.985354 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:29.985172    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:30.987945 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:30.987690    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:30.988361 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:30.987999    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:31.984771 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:31.984732    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:31.984954 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:31.984732    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:31.984954 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:31.984849    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:31.984954 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:31.984912    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:32.984560 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:32.984417    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:32.985007 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:32.984572    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:33.984976 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:33.984937    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:33.984976 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:33.984977    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:33.985497 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:33.985115    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:33.985497 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:33.985233    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:34.571019 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:34.570977    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:34.571247 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:34.571044    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs\") pod \"network-metrics-daemon-2c797\" (UID: \"b933b52f-75e9-478b-a95b-5cb77d236467\") " pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:34.571247 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:34.571106    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:34.571247 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:34.571179    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:34.571247 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:34.571197    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret podName:39f296d4-d2f6-46c7-ad0d-b520ecb0138e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:50.571172586 +0000 UTC m=+34.126328892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret") pod "global-pull-secret-syncer-cqpw2" (UID: "39f296d4-d2f6-46c7-ad0d-b520ecb0138e") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:34.571247 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:34.571235    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs podName:b933b52f-75e9-478b-a95b-5cb77d236467 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:50.57121698 +0000 UTC m=+34.126373270 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs") pod "network-metrics-daemon-2c797" (UID: "b933b52f-75e9-478b-a95b-5cb77d236467") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:34.772222 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:34.772187    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztf6\" (UniqueName: \"kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6\") pod \"network-check-target-2tx7l\" (UID: \"1761ac29-4d31-4d57-9355-5cffbdc2ef2c\") " pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:34.772400 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:34.772339    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:34.772400 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:34.772359    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:34.772400 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:34.772369    2557 projected.go:196] Error preparing data for projected volume kube-api-access-cztf6 for pod openshift-network-diagnostics/network-check-target-2tx7l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:34.772598 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:34.772431    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6 podName:1761ac29-4d31-4d57-9355-5cffbdc2ef2c nodeName:}" failed. No retries permitted until 2026-09-05 02:18:50.772413302 +0000 UTC m=+34.327569603 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cztf6" (UniqueName: "kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6") pod "network-check-target-2tx7l" (UID: "1761ac29-4d31-4d57-9355-5cffbdc2ef2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:34.984769 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:34.984690    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:34.984916 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:34.984829    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:35.985109 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:35.985077    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:35.985560 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:35.985214    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:35.985560 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:35.985241    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:35.985560 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:35.985353    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:36.985668 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:36.985631    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:36.986132 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:36.985750    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:37.984712 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:37.984676    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:37.984712 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:37.984705    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:37.984924 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:37.984843    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:37.984971 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:37.984917    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:38.985166 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:38.984932    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:38.986045 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:38.985221    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:39.171178 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.171142    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gq27j" event={"ID":"ad420093-4c18-44bf-b72b-ffa0c853997c","Type":"ContainerStarted","Data":"168144a3d5aae9d1fa38cd93f15b5b1961ef2984884b95dc5019790db46952d5"}
Sep 05 02:18:39.172718 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.172692    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lpd6d" event={"ID":"2836c4ea-8e45-4bd3-9c96-ace613f40ed2","Type":"ContainerStarted","Data":"648dbedc215a5ea507ce7ec184172066334ba1c8c9daf65b811e6778cdd779c8"}
Sep 05 02:18:39.174236 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.174209    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj" event={"ID":"a85eb80b-fc49-442c-96a8-9d4125c8044a","Type":"ContainerStarted","Data":"73bc4eb85c91261d81bbdb0b5d04910c9bf92fb606a9961c5390468951a5bb02"}
Sep 05 02:18:39.175859 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.175837    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-925z5" event={"ID":"9a0adfb7-0d26-4292-8790-81e38ba82c1a","Type":"ContainerStarted","Data":"67348a756803eda368346e65fee9e50d3392d875814e925f9d42998b7bfc6d88"}
Sep 05 02:18:39.177469 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.177431    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g8xt5" event={"ID":"c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b","Type":"ContainerStarted","Data":"92c19604209809802397cbd170611cbc218881b3878aa26f8762c361516719d9"}
Sep 05 02:18:39.180721 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.180703    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp" event={"ID":"fee79b98-0963-4527-8484-07cfe3dae8d7","Type":"ContainerStarted","Data":"c7df6200910c1a3d567dc86e43a4422bcb7d316b09cf9d45ac40a08ea3d6051b"}
Sep 05 02:18:39.180818 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.180765    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp" event={"ID":"fee79b98-0963-4527-8484-07cfe3dae8d7","Type":"ContainerStarted","Data":"2786f4e2f369603a7022714ba0042c8fc6de2a0c3705daa505e071fa95fb6a34"}
Sep 05 02:18:39.180818 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.180810    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp" event={"ID":"fee79b98-0963-4527-8484-07cfe3dae8d7","Type":"ContainerStarted","Data":"7926c52a9fdd21672aefe5b25b6b4150368d81e5a9ef8203e3f2ed0ad94d70c7"}
Sep 05 02:18:39.180939 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.180843    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp" event={"ID":"fee79b98-0963-4527-8484-07cfe3dae8d7","Type":"ContainerStarted","Data":"34e953e54b6f549408197d4cf58a7d5332ea94a0323134ce9a7ea4c58d9585a5"}
Sep 05 02:18:39.180939 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.180870    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp" event={"ID":"fee79b98-0963-4527-8484-07cfe3dae8d7","Type":"ContainerStarted","Data":"d4d357a5e348be3bf7da0f79a55e2181361e599dc4c792fe2972a14e5cb8f5fa"}
Sep 05 02:18:39.180939 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.180905    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp" event={"ID":"fee79b98-0963-4527-8484-07cfe3dae8d7","Type":"ContainerStarted","Data":"686c75ea58b9458545429a8d2ab2ac4a4ea10a4699cec08384df7f2d77614161"}
Sep 05 02:18:39.182363 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.182342    2557 generic.go:356] "Generic (PLEG): container finished" podID="a78d01f0-d41b-40ad-8d73-ddc3df30e814" containerID="ede277b995eaf314cfc771fb203d1095eefc480835db8994bf54b1a6749107bd" exitCode=0
Sep 05 02:18:39.182476 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.182423    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j9fmm" event={"ID":"a78d01f0-d41b-40ad-8d73-ddc3df30e814","Type":"ContainerDied","Data":"ede277b995eaf314cfc771fb203d1095eefc480835db8994bf54b1a6749107bd"}
Sep 05 02:18:39.184022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.184004    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dvtdd" event={"ID":"43bb8d7d-c160-4ad9-85d5-6b07e88dfd31","Type":"ContainerStarted","Data":"5686ae4901402bfc01573e6de44779545840b3c7ad808dbc0553838ae9c8f04a"}
Sep 05 02:18:39.187954 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.187918    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gq27j" podStartSLOduration=4.000507873 podStartE2EDuration="22.187906652s" totalImagesPullingTime="18.187398779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:39.187695634 +0000 UTC m=+22.742851947" watchObservedRunningTime="2026-09-05 02:18:39.187906652 +0000 UTC m=+22.743062961"
Sep 05 02:18:39.188462 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.188401    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-128.ec2.internal" podStartSLOduration=21.188388463 podStartE2EDuration="21.188388463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:22.155329227 +0000 UTC m=+5.710485540" watchObservedRunningTime="2026-09-05 02:18:39.188388463 +0000 UTC m=+22.743544775"
Sep 05 02:18:39.215673 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.215626    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g8xt5" podStartSLOduration=4.179698354 podStartE2EDuration="22.215610255s" totalImagesPullingTime="18.035911901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:39.201831284 +0000 UTC m=+22.756987592" watchObservedRunningTime="2026-09-05 02:18:39.215610255 +0000 UTC m=+22.770766601"
Sep 05 02:18:39.230599 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.230544    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-925z5" podStartSLOduration=4.199349757 podStartE2EDuration="22.230527176s" totalImagesPullingTime="18.031177419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:39.230491649 +0000 UTC m=+22.785647959" watchObservedRunningTime="2026-09-05 02:18:39.230527176 +0000 UTC m=+22.785683486"
Sep 05 02:18:39.230875 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.230849    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lpd6d" podStartSLOduration=4.268549924 podStartE2EDuration="22.230841969s" totalImagesPullingTime="17.962292045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:39.21513974 +0000 UTC m=+22.770296050" watchObservedRunningTime="2026-09-05 02:18:39.230841969 +0000 UTC m=+22.785998277"
Sep 05 02:18:39.245418 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.245317    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dvtdd" podStartSLOduration=11.586036151 podStartE2EDuration="22.245301575s" totalImagesPullingTime="10.659265424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:39.244622991 +0000 UTC m=+22.799779299" watchObservedRunningTime="2026-09-05 02:18:39.245301575 +0000 UTC m=+22.800457884"
Sep 05 02:18:39.760918 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.760894    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 05 02:18:39.886218 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.885895    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-05T02:18:39.76091321Z","UUID":"5f7d935c-de72-44ba-ac1d-6d984c65c63a","Handler":null,"Name":"","Endpoint":""}
Sep 05 02:18:39.887096 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.887077    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 05 02:18:39.887214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.887114    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 05 02:18:39.984526 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.984494    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:39.984639 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:39.984494    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:39.984639 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:39.984600    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:39.984737 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:39.984660    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:40.188241 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:40.188150    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6c7pg" event={"ID":"655a8af3-cdba-4d83-b644-a2f072ec2176","Type":"ContainerStarted","Data":"6881f318b1b94352627348be10ddbd82f2daea184a71127e5efd6ec24a936666"}
Sep 05 02:18:40.190331 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:40.190301    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj" event={"ID":"a85eb80b-fc49-442c-96a8-9d4125c8044a","Type":"ContainerStarted","Data":"ea89deae6c1fdf86da87dbc67f14d787d4867362013e63cf351ec17f6c83619a"}
Sep 05 02:18:40.984709 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:40.984678    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:40.984826 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:40.984808    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:41.195058 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:41.195018    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj" event={"ID":"a85eb80b-fc49-442c-96a8-9d4125c8044a","Type":"ContainerStarted","Data":"cbac2d790e16a378660901d71588ca558478c58f574982fcdce506ec83cb3dd9"}
Sep 05 02:18:41.199585 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:41.199558    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp" event={"ID":"fee79b98-0963-4527-8484-07cfe3dae8d7","Type":"ContainerStarted","Data":"4109f4ba67c678e097a951a84f88be63042b97988f0fac8e11c08f3ea2bfface"}
Sep 05 02:18:41.212572 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:41.212526    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6c7pg" podStartSLOduration=6.209966219 podStartE2EDuration="24.212507593s" totalImagesPullingTime="18.002541374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:40.204109987 +0000 UTC m=+23.759266291" watchObservedRunningTime="2026-09-05 02:18:41.212507593 +0000 UTC m=+24.767663901"
Sep 05 02:18:41.212846 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:41.212819    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qnpj" podStartSLOduration=3.8607999250000002 podStartE2EDuration="24.212811837s" totalImagesPullingTime="20.352011912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:41.21223585 +0000 UTC m=+24.767392162" watchObservedRunningTime="2026-09-05 02:18:41.212811837 +0000 UTC m=+24.767968145"
Sep 05 02:18:41.271755 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:41.271724    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dvtdd"
Sep 05 02:18:41.272269 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:41.272249    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dvtdd"
Sep 05 02:18:41.985202 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:41.985176    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:41.985355 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:41.985278    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:41.985355 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:41.985333    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:41.985427 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:41.985410    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:42.202315 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:42.202286    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dvtdd"
Sep 05 02:18:42.202777 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:42.202741    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dvtdd"
Sep 05 02:18:42.985140 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:42.985114    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:42.985318 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:42.985230    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:43.985589 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:43.985323    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:43.985917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:43.985353    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:43.985917 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:43.985725    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:43.985917 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:43.985835    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:44.209288 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:44.209205    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp" event={"ID":"fee79b98-0963-4527-8484-07cfe3dae8d7","Type":"ContainerStarted","Data":"8d59d528a2642dc0649fff5c883c32fdb7fba17502ac128e77b28cccef3e7228"}
Sep 05 02:18:44.209433 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:44.209412    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:44.211291 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:44.211265    2557 generic.go:356] "Generic (PLEG): container finished" podID="a78d01f0-d41b-40ad-8d73-ddc3df30e814" containerID="aea61a6066f96ffa2143e035cb30f27f772bfba697b8b3776c252f5d6c38c18a" exitCode=0
Sep 05 02:18:44.211391 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:44.211350    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j9fmm" event={"ID":"a78d01f0-d41b-40ad-8d73-ddc3df30e814","Type":"ContainerDied","Data":"aea61a6066f96ffa2143e035cb30f27f772bfba697b8b3776c252f5d6c38c18a"}
Sep 05 02:18:44.224558 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:44.224538    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:44.236548 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:44.235547    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp" podStartSLOduration=9.2085009 podStartE2EDuration="27.235526816s" totalImagesPullingTime="18.027025916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:44.234509719 +0000 UTC m=+27.789666052" watchObservedRunningTime="2026-09-05 02:18:44.235526816 +0000 UTC m=+27.790683128"
Sep 05 02:18:44.984738 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:44.984700    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:44.985002 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:44.984846    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:45.214390 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:45.214348    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:45.214989 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:45.214474    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:45.230318 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:45.230294    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:18:45.654344 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:45.654277    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2tx7l"]
Sep 05 02:18:45.654488 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:45.654410    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:45.654549 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:45.654508    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:45.657167 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:45.657138    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cqpw2"]
Sep 05 02:18:45.657275 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:45.657246    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:45.657351 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:45.657328    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:45.657720 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:45.657696    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2c797"]
Sep 05 02:18:45.657818 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:45.657796    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:45.657910 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:45.657888    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:46.217820 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:46.217788    2557 generic.go:356] "Generic (PLEG): container finished" podID="a78d01f0-d41b-40ad-8d73-ddc3df30e814" containerID="a3d5ef319c6a0139275331e4a2cc66a444163063a67f6b6032cb4ded8828cef1" exitCode=0
Sep 05 02:18:46.218530 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:46.217877    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j9fmm" event={"ID":"a78d01f0-d41b-40ad-8d73-ddc3df30e814","Type":"ContainerDied","Data":"a3d5ef319c6a0139275331e4a2cc66a444163063a67f6b6032cb4ded8828cef1"}
Sep 05 02:18:46.986141 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:46.985947    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:46.986325 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:46.986001    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:46.986325 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:46.986244    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:46.986325 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:46.986305    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:47.836840 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:47.836806    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:47.930266 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:47.930224    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:47.985093 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:47.985065    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:47.985211 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:47.985159    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:48.225103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:48.225070    2557 generic.go:356] "Generic (PLEG): container finished" podID="a78d01f0-d41b-40ad-8d73-ddc3df30e814" containerID="87f9bd0de4bb3ab43ffa2f878db73cc429c468bc9a8dc27b513e8faad071f7c6" exitCode=0
Sep 05 02:18:48.225268 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:48.225118    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j9fmm" event={"ID":"a78d01f0-d41b-40ad-8d73-ddc3df30e814","Type":"ContainerDied","Data":"87f9bd0de4bb3ab43ffa2f878db73cc429c468bc9a8dc27b513e8faad071f7c6"}
Sep 05 02:18:48.984888 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:48.984857    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:48.985332 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:48.984965    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:48.985332 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:48.985020    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:48.985332 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:48.985104    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:49.868097 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:49.868015    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:49.929517 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:49.929478    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:49.984846 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:49.984815    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:49.985010 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:49.984936    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:50.010265 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:50.010233    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:50.596781 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:50.596746    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:50.597042 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:50.596792    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs\") pod \"network-metrics-daemon-2c797\" (UID: \"b933b52f-75e9-478b-a95b-5cb77d236467\") " pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:50.597042 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:50.596887    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:50.597042 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:50.596938    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:50.597042 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:50.596955    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs podName:b933b52f-75e9-478b-a95b-5cb77d236467 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:22.596924651 +0000 UTC m=+66.152080941 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs") pod "network-metrics-daemon-2c797" (UID: "b933b52f-75e9-478b-a95b-5cb77d236467") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:50.597042 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:50.596995    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret podName:39f296d4-d2f6-46c7-ad0d-b520ecb0138e nodeName:}" failed. No retries permitted until 2026-09-05 02:19:22.596978014 +0000 UTC m=+66.152134304 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret") pod "global-pull-secret-syncer-cqpw2" (UID: "39f296d4-d2f6-46c7-ad0d-b520ecb0138e") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:50.797467 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:50.797415    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztf6\" (UniqueName: \"kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6\") pod \"network-check-target-2tx7l\" (UID: \"1761ac29-4d31-4d57-9355-5cffbdc2ef2c\") " pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:50.797656 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:50.797566    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:50.797656 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:50.797589    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:50.797656 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:50.797601    2557 projected.go:196] Error preparing data for projected volume kube-api-access-cztf6 for pod openshift-network-diagnostics/network-check-target-2tx7l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:50.797656 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:50.797654    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6 podName:1761ac29-4d31-4d57-9355-5cffbdc2ef2c nodeName:}" failed. No retries permitted until 2026-09-05 02:19:22.797640029 +0000 UTC m=+66.352796317 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cztf6" (UniqueName: "kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6") pod "network-check-target-2tx7l" (UID: "1761ac29-4d31-4d57-9355-5cffbdc2ef2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:50.984628 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:50.984547    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:50.984782 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:50.984548    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:50.984782 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:50.984709    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2c797" podUID="b933b52f-75e9-478b-a95b-5cb77d236467"
Sep 05 02:18:50.984782 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:50.984746    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqpw2" podUID="39f296d4-d2f6-46c7-ad0d-b520ecb0138e"
Sep 05 02:18:51.985279 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:51.985203    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:51.985703 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:51.985332    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tx7l" podUID="1761ac29-4d31-4d57-9355-5cffbdc2ef2c"
Sep 05 02:18:52.218875 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.218736    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-128.ec2.internal" event="NodeReady"
Sep 05 02:18:52.218875 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.218877    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 05 02:18:52.277150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.277059    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qs48v"]
Sep 05 02:18:52.280936 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.280913    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:52.284165 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.284119    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 05 02:18:52.284295 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.284205    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 05 02:18:52.284363 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.284317    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k9hcm\""
Sep 05 02:18:52.287262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.287240    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5t47h"]
Sep 05 02:18:52.290214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.290195    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5t47h"
Sep 05 02:18:52.293257 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.293131    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 05 02:18:52.293771 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.293752    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tl8b6\""
Sep 05 02:18:52.293867 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.293833    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 05 02:18:52.293988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.293960    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 05 02:18:52.296928 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.295175    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qs48v"]
Sep 05 02:18:52.301404 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.301381    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5t47h"]
Sep 05 02:18:52.408217 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.408184    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19ece4dc-ca43-42a0-bf9c-64ab7e233025-config-volume\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:52.408386 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.408237    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:52.408386 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.408259    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7310203-e7ca-467e-945c-d4a9db60e8fd-cert\") pod \"ingress-canary-5t47h\" (UID: \"c7310203-e7ca-467e-945c-d4a9db60e8fd\") " pod="openshift-ingress-canary/ingress-canary-5t47h"
Sep 05 02:18:52.408386 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.408355    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz97z\" (UniqueName: \"kubernetes.io/projected/c7310203-e7ca-467e-945c-d4a9db60e8fd-kube-api-access-qz97z\") pod \"ingress-canary-5t47h\" (UID: \"c7310203-e7ca-467e-945c-d4a9db60e8fd\") " pod="openshift-ingress-canary/ingress-canary-5t47h"
Sep 05 02:18:52.408552 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.408398    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ece4dc-ca43-42a0-bf9c-64ab7e233025-tmp-dir\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:52.408552 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.408455    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvd8x\" (UniqueName: \"kubernetes.io/projected/19ece4dc-ca43-42a0-bf9c-64ab7e233025-kube-api-access-bvd8x\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:52.508852 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.508813    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvd8x\" (UniqueName: \"kubernetes.io/projected/19ece4dc-ca43-42a0-bf9c-64ab7e233025-kube-api-access-bvd8x\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:52.509035 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.508864    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19ece4dc-ca43-42a0-bf9c-64ab7e233025-config-volume\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:52.509035 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.509011    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:52.509140 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.509057    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7310203-e7ca-467e-945c-d4a9db60e8fd-cert\") pod \"ingress-canary-5t47h\" (UID: \"c7310203-e7ca-467e-945c-d4a9db60e8fd\") " pod="openshift-ingress-canary/ingress-canary-5t47h"
Sep 05 02:18:52.509187 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.509136    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz97z\" (UniqueName: \"kubernetes.io/projected/c7310203-e7ca-467e-945c-d4a9db60e8fd-kube-api-access-qz97z\") pod \"ingress-canary-5t47h\" (UID: \"c7310203-e7ca-467e-945c-d4a9db60e8fd\") " pod="openshift-ingress-canary/ingress-canary-5t47h"
Sep 05 02:18:52.509241 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.509181    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ece4dc-ca43-42a0-bf9c-64ab7e233025-tmp-dir\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:52.509294 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:52.509262    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:52.509346 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:52.509323    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls podName:19ece4dc-ca43-42a0-bf9c-64ab7e233025 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:53.009306709 +0000 UTC m=+36.564463011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls") pod "dns-default-qs48v" (UID: "19ece4dc-ca43-42a0-bf9c-64ab7e233025") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:52.509508 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.509478    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ece4dc-ca43-42a0-bf9c-64ab7e233025-tmp-dir\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:52.509602 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.509531    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19ece4dc-ca43-42a0-bf9c-64ab7e233025-config-volume\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:52.513839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.513818    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7310203-e7ca-467e-945c-d4a9db60e8fd-cert\") pod \"ingress-canary-5t47h\" (UID: \"c7310203-e7ca-467e-945c-d4a9db60e8fd\") " pod="openshift-ingress-canary/ingress-canary-5t47h"
Sep 05 02:18:52.519431 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.519268    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvd8x\" (UniqueName: \"kubernetes.io/projected/19ece4dc-ca43-42a0-bf9c-64ab7e233025-kube-api-access-bvd8x\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:52.519521 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.519501    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz97z\" (UniqueName: \"kubernetes.io/projected/c7310203-e7ca-467e-945c-d4a9db60e8fd-kube-api-access-qz97z\") pod \"ingress-canary-5t47h\" (UID: \"c7310203-e7ca-467e-945c-d4a9db60e8fd\") " pod="openshift-ingress-canary/ingress-canary-5t47h"
Sep 05 02:18:52.603544 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.603513    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5t47h"
Sep 05 02:18:52.984511 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.984486    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:18:52.984629 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.984518    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:18:52.987453 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.987420    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zqlb9\""
Sep 05 02:18:52.988022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.987769    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:18:52.988080 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:52.988038    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:18:53.012573 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:53.012551    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:53.012710 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:53.012694    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:53.012768 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:53.012757    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls podName:19ece4dc-ca43-42a0-bf9c-64ab7e233025 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:54.012739381 +0000 UTC m=+37.567895677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls") pod "dns-default-qs48v" (UID: "19ece4dc-ca43-42a0-bf9c-64ab7e233025") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:53.100203 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:53.100168    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5t47h"]
Sep 05 02:18:53.113269 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:18:53.113243    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7310203_e7ca_467e_945c_d4a9db60e8fd.slice/crio-6732d5b84348ff2547109f177ec45dcbee4a6666448f4c6b97f61902c914fb53 WatchSource:0}: Error finding container 6732d5b84348ff2547109f177ec45dcbee4a6666448f4c6b97f61902c914fb53: Status 404 returned error can't find the container with id 6732d5b84348ff2547109f177ec45dcbee4a6666448f4c6b97f61902c914fb53
Sep 05 02:18:53.237030 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:53.236994    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5t47h" event={"ID":"c7310203-e7ca-467e-945c-d4a9db60e8fd","Type":"ContainerStarted","Data":"6732d5b84348ff2547109f177ec45dcbee4a6666448f4c6b97f61902c914fb53"}
Sep 05 02:18:53.239778 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:53.239755    2557 generic.go:356] "Generic (PLEG): container finished" podID="a78d01f0-d41b-40ad-8d73-ddc3df30e814" containerID="1fecefe915fec798cadb53480d04d8e41184f3a40ca4d5b89fbbc18b4b26f118" exitCode=0
Sep 05 02:18:53.239883 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:53.239795    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j9fmm" event={"ID":"a78d01f0-d41b-40ad-8d73-ddc3df30e814","Type":"ContainerDied","Data":"1fecefe915fec798cadb53480d04d8e41184f3a40ca4d5b89fbbc18b4b26f118"}
Sep 05 02:18:53.984274 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:53.984240    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:18:53.987017 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:53.986991    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:18:53.987521 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:53.987500    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:18:53.987977 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:53.987542    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rhfc\""
Sep 05 02:18:54.019807 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:54.019783    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:54.019943 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:54.019929    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:54.019996 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:54.019987    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls podName:19ece4dc-ca43-42a0-bf9c-64ab7e233025 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:56.019969876 +0000 UTC m=+39.575126177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls") pod "dns-default-qs48v" (UID: "19ece4dc-ca43-42a0-bf9c-64ab7e233025") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:54.248373 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:54.248292    2557 generic.go:356] "Generic (PLEG): container finished" podID="a78d01f0-d41b-40ad-8d73-ddc3df30e814" containerID="07cc759669d6562a9f74a757c0557296dc4b2a66cb7023268004a97d489b44db" exitCode=0
Sep 05 02:18:54.248373 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:54.248350    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j9fmm" event={"ID":"a78d01f0-d41b-40ad-8d73-ddc3df30e814","Type":"ContainerDied","Data":"07cc759669d6562a9f74a757c0557296dc4b2a66cb7023268004a97d489b44db"}
Sep 05 02:18:55.253667 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:55.253635    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j9fmm" event={"ID":"a78d01f0-d41b-40ad-8d73-ddc3df30e814","Type":"ContainerStarted","Data":"b115c406d51fcede2cd4cc8bd680695b8dd6155c6054578446c12654f42b73c8"}
Sep 05 02:18:55.279389 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:55.279331    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j9fmm" podStartSLOduration=8.788585505 podStartE2EDuration="38.279314508s" totalImagesPullingTime="29.490729003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:55.278217555 +0000 UTC m=+38.833373864" watchObservedRunningTime="2026-09-05 02:18:55.279314508 +0000 UTC m=+38.834470818"
Sep 05 02:18:55.400368 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:55.400263    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:55.500815 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:55.500780    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:55.742243 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:55.742204    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:56.034260 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:56.034175    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:18:56.034399 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:56.034333    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:56.034457 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:18:56.034399    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls podName:19ece4dc-ca43-42a0-bf9c-64ab7e233025 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:00.034383724 +0000 UTC m=+43.589540011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls") pod "dns-default-qs48v" (UID: "19ece4dc-ca43-42a0-bf9c-64ab7e233025") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:56.258044 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:56.258005    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5t47h" event={"ID":"c7310203-e7ca-467e-945c-d4a9db60e8fd","Type":"ContainerStarted","Data":"3be2357d2a19a9ab180e2258854a9cd29f016c96889a333fcfe2d641c779efdf"}
Sep 05 02:18:56.275648 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:56.275602    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5t47h" podStartSLOduration=1.80413307 podStartE2EDuration="4.275588589s" totalImagesPullingTime="2.471455519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:56.274857131 +0000 UTC m=+39.830013440" watchObservedRunningTime="2026-09-05 02:18:56.275588589 +0000 UTC m=+39.830744894"
Sep 05 02:18:58.348811 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:58.348767    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:18:58.449651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:18:58.449618    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:00.059162 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:00.059123    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:19:00.059585 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:00.059233    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:19:00.059585 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:00.059291    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls podName:19ece4dc-ca43-42a0-bf9c-64ab7e233025 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:08.059275852 +0000 UTC m=+51.614432139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls") pod "dns-default-qs48v" (UID: "19ece4dc-ca43-42a0-bf9c-64ab7e233025") : secret "dns-default-metrics-tls" not found
Sep 05 02:19:00.076759 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:00.076730    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:00.150008 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:00.149967    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:08.117850 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:08.117804    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:19:08.118272 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:08.117954    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:19:08.118272 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:08.118029    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls podName:19ece4dc-ca43-42a0-bf9c-64ab7e233025 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:24.118014173 +0000 UTC m=+67.673170463 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls") pod "dns-default-qs48v" (UID: "19ece4dc-ca43-42a0-bf9c-64ab7e233025") : secret "dns-default-metrics-tls" not found
Sep 05 02:19:10.865525 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:10.865483    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:10.974410 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:10.974378    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:17.244433 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:17.244405    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4kmp"
Sep 05 02:19:17.408989 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:17.408954    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:18.773981 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:18.773941    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:20.323575 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:20.323537    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:20.393990 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:20.393960    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:22.615399 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:22.615355    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs\") pod \"network-metrics-daemon-2c797\" (UID: \"b933b52f-75e9-478b-a95b-5cb77d236467\") " pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:19:22.615779 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:22.615423    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:19:22.621155 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:22.621135    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:19:22.621238 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:22.621202    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:19:22.626059 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:22.626041    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 05 02:19:22.626142 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:22.626111    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs podName:b933b52f-75e9-478b-a95b-5cb77d236467 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:26.626093051 +0000 UTC m=+130.181249339 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs") pod "network-metrics-daemon-2c797" (UID: "b933b52f-75e9-478b-a95b-5cb77d236467") : secret "metrics-daemon-secret" not found
Sep 05 02:19:22.628849 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:22.628831    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f296d4-d2f6-46c7-ad0d-b520ecb0138e-original-pull-secret\") pod \"global-pull-secret-syncer-cqpw2\" (UID: \"39f296d4-d2f6-46c7-ad0d-b520ecb0138e\") " pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:19:22.700094 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:22.700069    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqpw2"
Sep 05 02:19:22.817294 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:22.817264    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztf6\" (UniqueName: \"kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6\") pod \"network-check-target-2tx7l\" (UID: \"1761ac29-4d31-4d57-9355-5cffbdc2ef2c\") " pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:19:22.821332 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:22.821315    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:19:22.831117 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:22.831097    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:19:22.840617 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:22.840598    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cztf6\" (UniqueName: \"kubernetes.io/projected/1761ac29-4d31-4d57-9355-5cffbdc2ef2c-kube-api-access-cztf6\") pod \"network-check-target-2tx7l\" (UID: \"1761ac29-4d31-4d57-9355-5cffbdc2ef2c\") " pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:19:23.000053 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:22.999980    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cqpw2"]
Sep 05 02:19:23.002838 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:19:23.002805    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f296d4_d2f6_46c7_ad0d_b520ecb0138e.slice/crio-ec6eaf9806540bf6ba0876c2135c5f7be359141ce88c25deca203e4bb29535c0 WatchSource:0}: Error finding container ec6eaf9806540bf6ba0876c2135c5f7be359141ce88c25deca203e4bb29535c0: Status 404 returned error can't find the container with id ec6eaf9806540bf6ba0876c2135c5f7be359141ce88c25deca203e4bb29535c0
Sep 05 02:19:23.098872 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:23.098843    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rhfc\""
Sep 05 02:19:23.105886 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:23.105865    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:19:23.323592 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:23.323283    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cqpw2" event={"ID":"39f296d4-d2f6-46c7-ad0d-b520ecb0138e","Type":"ContainerStarted","Data":"ec6eaf9806540bf6ba0876c2135c5f7be359141ce88c25deca203e4bb29535c0"}
Sep 05 02:19:23.403228 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:23.403195    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2tx7l"]
Sep 05 02:19:23.405774 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:19:23.405747    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1761ac29_4d31_4d57_9355_5cffbdc2ef2c.slice/crio-b051eb8f404b42defee8812dec363d2ad5d73934af8e8278dd06eefd993c54ec WatchSource:0}: Error finding container b051eb8f404b42defee8812dec363d2ad5d73934af8e8278dd06eefd993c54ec: Status 404 returned error can't find the container with id b051eb8f404b42defee8812dec363d2ad5d73934af8e8278dd06eefd993c54ec
Sep 05 02:19:24.125749 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:24.125706    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:19:24.126254 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:24.125855    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:19:24.126254 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:24.125923    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls podName:19ece4dc-ca43-42a0-bf9c-64ab7e233025 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:56.125903656 +0000 UTC m=+99.681059954 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls") pod "dns-default-qs48v" (UID: "19ece4dc-ca43-42a0-bf9c-64ab7e233025") : secret "dns-default-metrics-tls" not found
Sep 05 02:19:24.328707 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:24.328670    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2tx7l" event={"ID":"1761ac29-4d31-4d57-9355-5cffbdc2ef2c","Type":"ContainerStarted","Data":"b051eb8f404b42defee8812dec363d2ad5d73934af8e8278dd06eefd993c54ec"}
Sep 05 02:19:24.869475 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:24.869399    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:24.927547 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:24.927505    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:28.340055 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:28.340011    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cqpw2" event={"ID":"39f296d4-d2f6-46c7-ad0d-b520ecb0138e","Type":"ContainerStarted","Data":"aac236e82754fd11357f599b1961c663c4cfacf9cefcf547b5dc9d834dadb0db"}
Sep 05 02:19:28.341473 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:28.341427    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2tx7l" event={"ID":"1761ac29-4d31-4d57-9355-5cffbdc2ef2c","Type":"ContainerStarted","Data":"6e382c9f3ec94558d789599d9e6743e38bdd80cad92c5609162b4ab85fa91ac2"}
Sep 05 02:19:28.341663 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:28.341559    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:19:28.354971 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:28.354922    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cqpw2" podStartSLOduration=66.696573788 podStartE2EDuration="1m11.354907279s" totalImagesPullingTime="4.658333491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:28.354425194 +0000 UTC m=+71.909581507" watchObservedRunningTime="2026-09-05 02:19:28.354907279 +0000 UTC m=+71.910063587"
Sep 05 02:19:28.370800 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:28.370756    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2tx7l" podStartSLOduration=67.118204919 podStartE2EDuration="1m11.370736992s" totalImagesPullingTime="4.252532073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:28.370681439 +0000 UTC m=+71.925837748" watchObservedRunningTime="2026-09-05 02:19:28.370736992 +0000 UTC m=+71.925893291"
Sep 05 02:19:56.141585 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.141546    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:19:56.141994 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:56.141658    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:19:56.141994 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:56.141723    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls podName:19ece4dc-ca43-42a0-bf9c-64ab7e233025 nodeName:}" failed. No retries permitted until 2026-09-05 02:21:00.141708198 +0000 UTC m=+163.696864486 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls") pod "dns-default-qs48v" (UID: "19ece4dc-ca43-42a0-bf9c-64ab7e233025") : secret "dns-default-metrics-tls" not found
Sep 05 02:19:56.399533 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.399451    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-ckvsw"]
Sep 05 02:19:56.403541 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.403520    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-ckvsw"
Sep 05 02:19:56.408317 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.408291    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vqgv5\""
Sep 05 02:19:56.418665 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.418640    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-ckvsw"]
Sep 05 02:19:56.443742 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.443717    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxx68\" (UniqueName: \"kubernetes.io/projected/7d0c0880-e395-4aec-911f-234bf4f37f28-kube-api-access-wxx68\") pod \"network-check-source-685456655f-ckvsw\" (UID: \"7d0c0880-e395-4aec-911f-234bf4f37f28\") " pod="openshift-network-diagnostics/network-check-source-685456655f-ckvsw"
Sep 05 02:19:56.500544 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.500514    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4"]
Sep 05 02:19:56.503107 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.503091    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4"
Sep 05 02:19:56.505294 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.505264    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 05 02:19:56.505405 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.505280    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:19:56.505405 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.505278    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 05 02:19:56.505604 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.505590    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 05 02:19:56.505686 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.505668    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-h5swv\""
Sep 05 02:19:56.518874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.518847    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-tnh5c"]
Sep 05 02:19:56.521591 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.521572    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6754f8dbcf-m2qqr"]
Sep 05 02:19:56.521737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.521716    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.524082 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.524062    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4"]
Sep 05 02:19:56.524177 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.524163    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.527668 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.527647    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 05 02:19:56.527800 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.527781    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 05 02:19:56.528080 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.528067    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 05 02:19:56.528169 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.528106    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-v488k\""
Sep 05 02:19:56.528228 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.528182    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 05 02:19:56.528228 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.528200    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 05 02:19:56.528584 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.528572    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 05 02:19:56.531720 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.531702    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 05 02:19:56.532086 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.532069    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hnv4q\""
Sep 05 02:19:56.541418 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.541390    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-tnh5c"]
Sep 05 02:19:56.543493 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.543472    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 05 02:19:56.544690 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.544663    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12667c29-3724-4078-964d-07268713477a-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-54qc4\" (UID: \"12667c29-3724-4078-964d-07268713477a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4"
Sep 05 02:19:56.544808 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.544705    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d16b566d-44e4-46fe-bf78-6bb6df91a954-snapshots\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.544808 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.544739    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d16b566d-44e4-46fe-bf78-6bb6df91a954-serving-cert\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.544808 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.544770    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjrct\" (UniqueName: \"kubernetes.io/projected/12667c29-3724-4078-964d-07268713477a-kube-api-access-jjrct\") pod \"kube-storage-version-migrator-operator-8d7dd4675-54qc4\" (UID: \"12667c29-3724-4078-964d-07268713477a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4"
Sep 05 02:19:56.545020 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.544814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16b566d-44e4-46fe-bf78-6bb6df91a954-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.545020 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.544878    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-bound-sa-token\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.545020 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.544911    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6754f8dbcf-m2qqr"]
Sep 05 02:19:56.545020 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.544934    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d16b566d-44e4-46fe-bf78-6bb6df91a954-tmp\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.545020 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.544986    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16b566d-44e4-46fe-bf78-6bb6df91a954-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.545248 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.545022    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxx68\" (UniqueName: \"kubernetes.io/projected/7d0c0880-e395-4aec-911f-234bf4f37f28-kube-api-access-wxx68\") pod \"network-check-source-685456655f-ckvsw\" (UID: \"7d0c0880-e395-4aec-911f-234bf4f37f28\") " pod="openshift-network-diagnostics/network-check-source-685456655f-ckvsw"
Sep 05 02:19:56.545248 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.545046    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4d1120c-2be6-4408-9862-61d45dc9625a-image-registry-private-configuration\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.545248 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.545064    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4d1120c-2be6-4408-9862-61d45dc9625a-trusted-ca\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.545248 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.545167    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4d1120c-2be6-4408-9862-61d45dc9625a-ca-trust-extracted\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.545248 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.545196    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-certificates\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.545248 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.545225    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12667c29-3724-4078-964d-07268713477a-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-54qc4\" (UID: \"12667c29-3724-4078-964d-07268713477a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4"
Sep 05 02:19:56.545457 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.545269    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf5jn\" (UniqueName: \"kubernetes.io/projected/d16b566d-44e4-46fe-bf78-6bb6df91a954-kube-api-access-gf5jn\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.545457 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.545292    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.545457 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.545318    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4d1120c-2be6-4408-9862-61d45dc9625a-installation-pull-secrets\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.545457 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.545356    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf8jr\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-kube-api-access-jf8jr\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.547032 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.547016    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 05 02:19:56.564039 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.564010    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxx68\" (UniqueName: \"kubernetes.io/projected/7d0c0880-e395-4aec-911f-234bf4f37f28-kube-api-access-wxx68\") pod \"network-check-source-685456655f-ckvsw\" (UID: \"7d0c0880-e395-4aec-911f-234bf4f37f28\") " pod="openshift-network-diagnostics/network-check-source-685456655f-ckvsw"
Sep 05 02:19:56.610040 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.610007    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"]
Sep 05 02:19:56.612916 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.612895    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-gtf6d"]
Sep 05 02:19:56.613033 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.613012    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"
Sep 05 02:19:56.615501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.615483    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:19:56.615979 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.615961    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 05 02:19:56.616317 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.616297    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 05 02:19:56.616419 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.616298    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:19:56.616419 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.616355    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pwns7\""
Sep 05 02:19:56.618379 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.618362    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:19:56.621172 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.621155    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 05 02:19:56.621396 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.621381    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-l54zc\""
Sep 05 02:19:56.621492 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.621464    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 05 02:19:56.621685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.621671    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 05 02:19:56.624284 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.624265    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 05 02:19:56.627213 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.627184    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"]
Sep 05 02:19:56.628354 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.628333    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-gtf6d"]
Sep 05 02:19:56.645695 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.645671    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.645790 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.645699    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4d1120c-2be6-4408-9862-61d45dc9625a-installation-pull-secrets\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.645790 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.645734    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf8jr\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-kube-api-access-jf8jr\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.645790 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.645761    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12667c29-3724-4078-964d-07268713477a-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-54qc4\" (UID: \"12667c29-3724-4078-964d-07268713477a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4"
Sep 05 02:19:56.645790 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.645787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83157c8a-294b-4add-aa27-3db843d0a9e5-trusted-ca\") pod \"console-operator-85b454874b-gtf6d\" (UID: \"83157c8a-294b-4add-aa27-3db843d0a9e5\") " pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:19:56.645978 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.645805    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jclq\" (UniqueName: \"kubernetes.io/projected/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-kube-api-access-7jclq\") pod \"cluster-samples-operator-7cdcfcf76c-gsk84\" (UID: \"265a7e5e-277b-4ffa-ab70-f4d5b3e65f84\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"
Sep 05 02:19:56.645978 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.645832    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjrct\" (UniqueName: \"kubernetes.io/projected/12667c29-3724-4078-964d-07268713477a-kube-api-access-jjrct\") pod \"kube-storage-version-migrator-operator-8d7dd4675-54qc4\" (UID: \"12667c29-3724-4078-964d-07268713477a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4"
Sep 05 02:19:56.645978 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:56.645894    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:19:56.645978 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:56.645911    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6754f8dbcf-m2qqr: secret "image-registry-tls" not found
Sep 05 02:19:56.645978 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.645942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83157c8a-294b-4add-aa27-3db843d0a9e5-config\") pod \"console-operator-85b454874b-gtf6d\" (UID: \"83157c8a-294b-4add-aa27-3db843d0a9e5\") " pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:19:56.646164 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:56.645978    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls podName:b4d1120c-2be6-4408-9862-61d45dc9625a nodeName:}" failed. No retries permitted until 2026-09-05 02:19:57.14595751 +0000 UTC m=+100.701113801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls") pod "image-registry-6754f8dbcf-m2qqr" (UID: "b4d1120c-2be6-4408-9862-61d45dc9625a") : secret "image-registry-tls" not found
Sep 05 02:19:56.646164 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.646054    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16b566d-44e4-46fe-bf78-6bb6df91a954-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.646164 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.646113    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4d1120c-2be6-4408-9862-61d45dc9625a-image-registry-private-configuration\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.646319 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.646209    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-certificates\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.646319 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.646249    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12667c29-3724-4078-964d-07268713477a-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-54qc4\" (UID: \"12667c29-3724-4078-964d-07268713477a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4"
Sep 05 02:19:56.646319 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.646310    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d16b566d-44e4-46fe-bf78-6bb6df91a954-snapshots\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.646487 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.646343    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d16b566d-44e4-46fe-bf78-6bb6df91a954-serving-cert\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.647131 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.647106    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16b566d-44e4-46fe-bf78-6bb6df91a954-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.647230 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.647148    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-bound-sa-token\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.647230 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.647180    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sx8z\" (UniqueName: \"kubernetes.io/projected/83157c8a-294b-4add-aa27-3db843d0a9e5-kube-api-access-7sx8z\") pod \"console-operator-85b454874b-gtf6d\" (UID: \"83157c8a-294b-4add-aa27-3db843d0a9e5\") " pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:19:56.647230 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.647039    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d16b566d-44e4-46fe-bf78-6bb6df91a954-snapshots\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.647230 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.647048    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16b566d-44e4-46fe-bf78-6bb6df91a954-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.647230 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.646772    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-certificates\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.647516 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.647023    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12667c29-3724-4078-964d-07268713477a-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-54qc4\" (UID: \"12667c29-3724-4078-964d-07268713477a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4"
Sep 05 02:19:56.647516 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.647313    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d16b566d-44e4-46fe-bf78-6bb6df91a954-tmp\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.647516 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.647346    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4d1120c-2be6-4408-9862-61d45dc9625a-trusted-ca\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.647516 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.647387    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4d1120c-2be6-4408-9862-61d45dc9625a-ca-trust-extracted\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.647516 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.647415    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gsk84\" (UID: \"265a7e5e-277b-4ffa-ab70-f4d5b3e65f84\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"
Sep 05 02:19:56.647516 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.647470    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83157c8a-294b-4add-aa27-3db843d0a9e5-serving-cert\") pod \"console-operator-85b454874b-gtf6d\" (UID: \"83157c8a-294b-4add-aa27-3db843d0a9e5\") " pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:19:56.647516 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.647502    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf5jn\" (UniqueName: \"kubernetes.io/projected/d16b566d-44e4-46fe-bf78-6bb6df91a954-kube-api-access-gf5jn\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.647868 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.647677    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16b566d-44e4-46fe-bf78-6bb6df91a954-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.647949 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.647925    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d16b566d-44e4-46fe-bf78-6bb6df91a954-tmp\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.648195 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.648175    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4d1120c-2be6-4408-9862-61d45dc9625a-ca-trust-extracted\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.648368 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.648346    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4d1120c-2be6-4408-9862-61d45dc9625a-trusted-ca\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.648670 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.648653    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12667c29-3724-4078-964d-07268713477a-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-54qc4\" (UID: \"12667c29-3724-4078-964d-07268713477a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4"
Sep 05 02:19:56.648851 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.648828    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4d1120c-2be6-4408-9862-61d45dc9625a-installation-pull-secrets\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.649053 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.649037    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4d1120c-2be6-4408-9862-61d45dc9625a-image-registry-private-configuration\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.649096 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.649082    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d16b566d-44e4-46fe-bf78-6bb6df91a954-serving-cert\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.655388 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.655338    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjrct\" (UniqueName: \"kubernetes.io/projected/12667c29-3724-4078-964d-07268713477a-kube-api-access-jjrct\") pod \"kube-storage-version-migrator-operator-8d7dd4675-54qc4\" (UID: \"12667c29-3724-4078-964d-07268713477a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4"
Sep 05 02:19:56.655600 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.655574    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf8jr\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-kube-api-access-jf8jr\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.656457 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.656423    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-bound-sa-token\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:56.659233 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.659212    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf5jn\" (UniqueName: \"kubernetes.io/projected/d16b566d-44e4-46fe-bf78-6bb6df91a954-kube-api-access-gf5jn\") pod \"insights-operator-76bcd6ccf5-tnh5c\" (UID: \"d16b566d-44e4-46fe-bf78-6bb6df91a954\") " pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.711921 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.711895    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-ckvsw"
Sep 05 02:19:56.733228 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.733193    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:56.748422 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.748387    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sx8z\" (UniqueName: \"kubernetes.io/projected/83157c8a-294b-4add-aa27-3db843d0a9e5-kube-api-access-7sx8z\") pod \"console-operator-85b454874b-gtf6d\" (UID: \"83157c8a-294b-4add-aa27-3db843d0a9e5\") " pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:19:56.748579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.748465    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gsk84\" (UID: \"265a7e5e-277b-4ffa-ab70-f4d5b3e65f84\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"
Sep 05 02:19:56.748579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.748504    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83157c8a-294b-4add-aa27-3db843d0a9e5-serving-cert\") pod \"console-operator-85b454874b-gtf6d\" (UID: \"83157c8a-294b-4add-aa27-3db843d0a9e5\") " pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:19:56.748579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.748565    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83157c8a-294b-4add-aa27-3db843d0a9e5-trusted-ca\") pod \"console-operator-85b454874b-gtf6d\" (UID: \"83157c8a-294b-4add-aa27-3db843d0a9e5\") " pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:19:56.748730 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.748595    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jclq\" (UniqueName: \"kubernetes.io/projected/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-kube-api-access-7jclq\") pod \"cluster-samples-operator-7cdcfcf76c-gsk84\" (UID: \"265a7e5e-277b-4ffa-ab70-f4d5b3e65f84\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"
Sep 05 02:19:56.748730 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:56.748617    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 05 02:19:56.748730 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.748630    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83157c8a-294b-4add-aa27-3db843d0a9e5-config\") pod \"console-operator-85b454874b-gtf6d\" (UID: \"83157c8a-294b-4add-aa27-3db843d0a9e5\") " pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:19:56.748730 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:56.748694    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls podName:265a7e5e-277b-4ffa-ab70-f4d5b3e65f84 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:57.248667108 +0000 UTC m=+100.803823432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gsk84" (UID: "265a7e5e-277b-4ffa-ab70-f4d5b3e65f84") : secret "samples-operator-tls" not found
Sep 05 02:19:56.749454 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.749409    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83157c8a-294b-4add-aa27-3db843d0a9e5-config\") pod \"console-operator-85b454874b-gtf6d\" (UID: \"83157c8a-294b-4add-aa27-3db843d0a9e5\") " pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:19:56.749606 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.749589    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83157c8a-294b-4add-aa27-3db843d0a9e5-trusted-ca\") pod \"console-operator-85b454874b-gtf6d\" (UID: \"83157c8a-294b-4add-aa27-3db843d0a9e5\") " pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:19:56.750999 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.750977    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83157c8a-294b-4add-aa27-3db843d0a9e5-serving-cert\") pod \"console-operator-85b454874b-gtf6d\" (UID: \"83157c8a-294b-4add-aa27-3db843d0a9e5\") " pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:19:56.760409 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.760286    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jclq\" (UniqueName: \"kubernetes.io/projected/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-kube-api-access-7jclq\") pod \"cluster-samples-operator-7cdcfcf76c-gsk84\" (UID: \"265a7e5e-277b-4ffa-ab70-f4d5b3e65f84\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"
Sep 05 02:19:56.763141 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.763119    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sx8z\" (UniqueName: \"kubernetes.io/projected/83157c8a-294b-4add-aa27-3db843d0a9e5-kube-api-access-7sx8z\") pod \"console-operator-85b454874b-gtf6d\" (UID: \"83157c8a-294b-4add-aa27-3db843d0a9e5\") " pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:19:56.811376 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.811344    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4"
Sep 05 02:19:56.831713 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.831685    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c"
Sep 05 02:19:56.840067 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.840035    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:56.930238 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.930150    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:19:56.978246 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:56.978214    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-ckvsw"]
Sep 05 02:19:56.983056 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:19:56.983026    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0c0880_e395_4aec_911f_234bf4f37f28.slice/crio-0356e2f4116dc52fed192885a10edfd36280030dc03eb6c1b246532970fb4a85 WatchSource:0}: Error finding container 0356e2f4116dc52fed192885a10edfd36280030dc03eb6c1b246532970fb4a85: Status 404 returned error can't find the container with id 0356e2f4116dc52fed192885a10edfd36280030dc03eb6c1b246532970fb4a85
Sep 05 02:19:57.083781 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:57.083727    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4"]
Sep 05 02:19:57.086588 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:19:57.086561    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12667c29_3724_4078_964d_07268713477a.slice/crio-9caea275f9a69b9009169c334aa509b4df412d03bedc24d4010a6999f307a99e WatchSource:0}: Error finding container 9caea275f9a69b9009169c334aa509b4df412d03bedc24d4010a6999f307a99e: Status 404 returned error can't find the container with id 9caea275f9a69b9009169c334aa509b4df412d03bedc24d4010a6999f307a99e
Sep 05 02:19:57.103738 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:57.103714    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-tnh5c"]
Sep 05 02:19:57.106729 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:19:57.106702    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16b566d_44e4_46fe_bf78_6bb6df91a954.slice/crio-5c83d80ef087e67c341bcdd1617d995fa1e2a1738b8c3d7d4911babf2225b9f9 WatchSource:0}: Error finding container 5c83d80ef087e67c341bcdd1617d995fa1e2a1738b8c3d7d4911babf2225b9f9: Status 404 returned error can't find the container with id 5c83d80ef087e67c341bcdd1617d995fa1e2a1738b8c3d7d4911babf2225b9f9
Sep 05 02:19:57.152073 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:57.152041    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:57.152342 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:57.152177    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:19:57.152342 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:57.152195    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6754f8dbcf-m2qqr: secret "image-registry-tls" not found
Sep 05 02:19:57.152342 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:57.152243    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls podName:b4d1120c-2be6-4408-9862-61d45dc9625a nodeName:}" failed. No retries permitted until 2026-09-05 02:19:58.152226942 +0000 UTC m=+101.707383245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls") pod "image-registry-6754f8dbcf-m2qqr" (UID: "b4d1120c-2be6-4408-9862-61d45dc9625a") : secret "image-registry-tls" not found
Sep 05 02:19:57.198318 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:57.198256    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-gtf6d"]
Sep 05 02:19:57.201188 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:19:57.201158    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83157c8a_294b_4add_aa27_3db843d0a9e5.slice/crio-4c6d24fde0ec45a663bd7c73818b67e78372e6fdce57ff1e63341a173e2254bd WatchSource:0}: Error finding container 4c6d24fde0ec45a663bd7c73818b67e78372e6fdce57ff1e63341a173e2254bd: Status 404 returned error can't find the container with id 4c6d24fde0ec45a663bd7c73818b67e78372e6fdce57ff1e63341a173e2254bd
Sep 05 02:19:57.252559 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:57.252532    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gsk84\" (UID: \"265a7e5e-277b-4ffa-ab70-f4d5b3e65f84\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"
Sep 05 02:19:57.252681 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:57.252663    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 05 02:19:57.252733 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:57.252718    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls podName:265a7e5e-277b-4ffa-ab70-f4d5b3e65f84 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:58.252705263 +0000 UTC m=+101.807861555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gsk84" (UID: "265a7e5e-277b-4ffa-ab70-f4d5b3e65f84") : secret "samples-operator-tls" not found
Sep 05 02:19:57.409485 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:57.409431    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4" event={"ID":"12667c29-3724-4078-964d-07268713477a","Type":"ContainerStarted","Data":"9caea275f9a69b9009169c334aa509b4df412d03bedc24d4010a6999f307a99e"}
Sep 05 02:19:57.411026 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:57.410993    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-ckvsw" event={"ID":"7d0c0880-e395-4aec-911f-234bf4f37f28","Type":"ContainerStarted","Data":"1e01c7dded279d4db07eed7c3040104f90a9e696f3ec27fa122b614f790a4382"}
Sep 05 02:19:57.411167 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:57.411047    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-ckvsw" event={"ID":"7d0c0880-e395-4aec-911f-234bf4f37f28","Type":"ContainerStarted","Data":"0356e2f4116dc52fed192885a10edfd36280030dc03eb6c1b246532970fb4a85"}
Sep 05 02:19:57.412177 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:57.412154    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-gtf6d" event={"ID":"83157c8a-294b-4add-aa27-3db843d0a9e5","Type":"ContainerStarted","Data":"4c6d24fde0ec45a663bd7c73818b67e78372e6fdce57ff1e63341a173e2254bd"}
Sep 05 02:19:57.413220 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:57.413202    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c" event={"ID":"d16b566d-44e4-46fe-bf78-6bb6df91a954","Type":"ContainerStarted","Data":"5c83d80ef087e67c341bcdd1617d995fa1e2a1738b8c3d7d4911babf2225b9f9"}
Sep 05 02:19:57.428238 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:57.428196    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-ckvsw" podStartSLOduration=1.4281836 podStartE2EDuration="1.4281836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:57.427102055 +0000 UTC m=+100.982258374" watchObservedRunningTime="2026-09-05 02:19:57.4281836 +0000 UTC m=+100.983339910"
Sep 05 02:19:58.160670 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:58.160013    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:19:58.160670 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:58.160248    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:19:58.160670 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:58.160263    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6754f8dbcf-m2qqr: secret "image-registry-tls" not found
Sep 05 02:19:58.160670 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:58.160318    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls podName:b4d1120c-2be6-4408-9862-61d45dc9625a nodeName:}" failed. No retries permitted until 2026-09-05 02:20:00.16030048 +0000 UTC m=+103.715456781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls") pod "image-registry-6754f8dbcf-m2qqr" (UID: "b4d1120c-2be6-4408-9862-61d45dc9625a") : secret "image-registry-tls" not found
Sep 05 02:19:58.261417 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:58.261381    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gsk84\" (UID: \"265a7e5e-277b-4ffa-ab70-f4d5b3e65f84\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"
Sep 05 02:19:58.261599 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:58.261559    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 05 02:19:58.261665 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:19:58.261624    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls podName:265a7e5e-277b-4ffa-ab70-f4d5b3e65f84 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:00.261610155 +0000 UTC m=+103.816766455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gsk84" (UID: "265a7e5e-277b-4ffa-ab70-f4d5b3e65f84") : secret "samples-operator-tls" not found
Sep 05 02:19:58.493168 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:58.492980    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:58.565489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:58.565382    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:19:59.346141 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:19:59.346111    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2tx7l"
Sep 05 02:20:00.178326 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:00.178297    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:20:00.178481 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:00.178429    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:20:00.178567 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:00.178484    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6754f8dbcf-m2qqr: secret "image-registry-tls" not found
Sep 05 02:20:00.178567 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:00.178541    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls podName:b4d1120c-2be6-4408-9862-61d45dc9625a nodeName:}" failed. No retries permitted until 2026-09-05 02:20:04.178519594 +0000 UTC m=+107.733675894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls") pod "image-registry-6754f8dbcf-m2qqr" (UID: "b4d1120c-2be6-4408-9862-61d45dc9625a") : secret "image-registry-tls" not found
Sep 05 02:20:00.283520 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:00.283327    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gsk84\" (UID: \"265a7e5e-277b-4ffa-ab70-f4d5b3e65f84\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"
Sep 05 02:20:00.283930 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:00.283909    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 05 02:20:00.284014 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:00.284002    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls podName:265a7e5e-277b-4ffa-ab70-f4d5b3e65f84 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:04.283979189 +0000 UTC m=+107.839135480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gsk84" (UID: "265a7e5e-277b-4ffa-ab70-f4d5b3e65f84") : secret "samples-operator-tls" not found
Sep 05 02:20:00.424646 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:00.424555    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c" event={"ID":"d16b566d-44e4-46fe-bf78-6bb6df91a954","Type":"ContainerStarted","Data":"a6ac8f09092d5678ac60b98c84511d589520d1ceb3d71160845249683071430a"}
Sep 05 02:20:00.426231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:00.426204    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4" event={"ID":"12667c29-3724-4078-964d-07268713477a","Type":"ContainerStarted","Data":"cb2558d992769f877b04edaccbcf6c3ff26aecc3c43c8d4fdb55a54692dbdb53"}
Sep 05 02:20:00.428050 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:00.428025    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/0.log"
Sep 05 02:20:00.428182 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:00.428070    2557 generic.go:356] "Generic (PLEG): container finished" podID="83157c8a-294b-4add-aa27-3db843d0a9e5" containerID="1cf1d367023c46ceeaf68df87d0ad14d621e66f55dc49ae95b914da58daefa0f" exitCode=255
Sep 05 02:20:00.428182 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:00.428098    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-gtf6d" event={"ID":"83157c8a-294b-4add-aa27-3db843d0a9e5","Type":"ContainerDied","Data":"1cf1d367023c46ceeaf68df87d0ad14d621e66f55dc49ae95b914da58daefa0f"}
Sep 05 02:20:00.428383 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:00.428367    2557 scope.go:117] "RemoveContainer" containerID="1cf1d367023c46ceeaf68df87d0ad14d621e66f55dc49ae95b914da58daefa0f"
Sep 05 02:20:00.442351 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:00.442311    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c" podStartSLOduration=1.436157343 podStartE2EDuration="4.442297742s" totalImagesPullingTime="3.006140399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:00.440969451 +0000 UTC m=+103.996125760" watchObservedRunningTime="2026-09-05 02:20:00.442297742 +0000 UTC m=+103.997454051"
Sep 05 02:20:00.473462 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:00.473204    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4" podStartSLOduration=1.452125926 podStartE2EDuration="4.47318434s" totalImagesPullingTime="3.021058414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:00.471747204 +0000 UTC m=+104.026903515" watchObservedRunningTime="2026-09-05 02:20:00.47318434 +0000 UTC m=+104.028340650"
Sep 05 02:20:01.048930 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.048884    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:01.159258 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.159224    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:01.186531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.186495    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sdzmq"]
Sep 05 02:20:01.202121 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.202100    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.204270 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.204244    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 05 02:20:01.205048 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.205011    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sdzmq"]
Sep 05 02:20:01.205358 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.205332    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mx6qs\""
Sep 05 02:20:01.206074 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.205848    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 05 02:20:01.248029 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.248008    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:01.293688 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.293653    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.293825 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.293698    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-crio-socket\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.293825 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.293763    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.293825 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.293809    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-data-volume\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.293927 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.293826    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5hf\" (UniqueName: \"kubernetes.io/projected/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-api-access-hv5hf\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.394945 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.394904    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.394945 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.394945    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-crio-socket\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.395140 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.395013    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-crio-socket\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.395140 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:01.395021    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:01.395140 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:01.395097    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm podName:b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b nodeName:}" failed. No retries permitted until 2026-09-05 02:20:01.895082304 +0000 UTC m=+105.450238595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdzmq" (UID: "b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:01.395140 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.395128    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.395290 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.395173    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-data-volume\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.395290 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.395190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5hf\" (UniqueName: \"kubernetes.io/projected/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-api-access-hv5hf\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.395358 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:01.395290    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:01.395358 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:01.395348    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls podName:b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b nodeName:}" failed. No retries permitted until 2026-09-05 02:20:01.895333546 +0000 UTC m=+105.450489846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdzmq" (UID: "b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:01.395539 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.395524    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-data-volume\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.403953 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.403928    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5hf\" (UniqueName: \"kubernetes.io/projected/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-api-access-hv5hf\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.433326 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.433300    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/1.log"
Sep 05 02:20:01.433741 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.433724    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/0.log"
Sep 05 02:20:01.433786 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.433765    2557 generic.go:356] "Generic (PLEG): container finished" podID="83157c8a-294b-4add-aa27-3db843d0a9e5" containerID="7166725e19a1bf66c499665a273faff39fcbab135ea8cebe00c65159d16e2a25" exitCode=255
Sep 05 02:20:01.433846 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.433823    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-gtf6d" event={"ID":"83157c8a-294b-4add-aa27-3db843d0a9e5","Type":"ContainerDied","Data":"7166725e19a1bf66c499665a273faff39fcbab135ea8cebe00c65159d16e2a25"}
Sep 05 02:20:01.433902 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.433893    2557 scope.go:117] "RemoveContainer" containerID="1cf1d367023c46ceeaf68df87d0ad14d621e66f55dc49ae95b914da58daefa0f"
Sep 05 02:20:01.434113 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.434059    2557 scope.go:117] "RemoveContainer" containerID="7166725e19a1bf66c499665a273faff39fcbab135ea8cebe00c65159d16e2a25"
Sep 05 02:20:01.434285 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:01.434260    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-gtf6d_openshift-console-operator(83157c8a-294b-4add-aa27-3db843d0a9e5)\"" pod="openshift-console-operator/console-operator-85b454874b-gtf6d" podUID="83157c8a-294b-4add-aa27-3db843d0a9e5"
Sep 05 02:20:01.582996 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.582963    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:01.658545 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.658466    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:01.898714 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.898673    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.898922 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:01.898743    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:01.898922 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:01.898827    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:01.898922 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:01.898909    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm podName:b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b nodeName:}" failed. No retries permitted until 2026-09-05 02:20:02.898888107 +0000 UTC m=+106.454044417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdzmq" (UID: "b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:01.898922 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:01.898835    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:01.899140 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:01.898969    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls podName:b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b nodeName:}" failed. No retries permitted until 2026-09-05 02:20:02.898955307 +0000 UTC m=+106.454111607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdzmq" (UID: "b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:02.269004 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:02.268968    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:02.438714 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:02.438688    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/1.log"
Sep 05 02:20:02.439086 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:02.438984    2557 scope.go:117] "RemoveContainer" containerID="7166725e19a1bf66c499665a273faff39fcbab135ea8cebe00c65159d16e2a25"
Sep 05 02:20:02.439154 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:02.439135    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-gtf6d_openshift-console-operator(83157c8a-294b-4add-aa27-3db843d0a9e5)\"" pod="openshift-console-operator/console-operator-85b454874b-gtf6d" podUID="83157c8a-294b-4add-aa27-3db843d0a9e5"
Sep 05 02:20:02.906402 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:02.906357    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:02.906595 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:02.906467    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:02.906595 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:02.906540    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:02.906595 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:02.906559    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:02.906696 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:02.906611    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm podName:b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b nodeName:}" failed. No retries permitted until 2026-09-05 02:20:04.906596637 +0000 UTC m=+108.461752937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdzmq" (UID: "b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:02.906696 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:02.906625    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls podName:b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b nodeName:}" failed. No retries permitted until 2026-09-05 02:20:04.906619914 +0000 UTC m=+108.461776201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdzmq" (UID: "b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:03.546283 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:03.546248    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:03.609857 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:03.609818    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:03.899730 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:03.899688    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:03.968952 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:03.968917    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:04.145819 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:04.145791    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g8xt5_c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b/dns-node-resolver/0.log"
Sep 05 02:20:04.217013 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:04.216942    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:20:04.217136 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:04.217097    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:20:04.217136 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:04.217116    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6754f8dbcf-m2qqr: secret "image-registry-tls" not found
Sep 05 02:20:04.217205 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:04.217170    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls podName:b4d1120c-2be6-4408-9862-61d45dc9625a nodeName:}" failed. No retries permitted until 2026-09-05 02:20:12.217155416 +0000 UTC m=+115.772311708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls") pod "image-registry-6754f8dbcf-m2qqr" (UID: "b4d1120c-2be6-4408-9862-61d45dc9625a") : secret "image-registry-tls" not found
Sep 05 02:20:04.318404 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:04.318348    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gsk84\" (UID: \"265a7e5e-277b-4ffa-ab70-f4d5b3e65f84\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"
Sep 05 02:20:04.318624 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:04.318539    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 05 02:20:04.318624 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:04.318621    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls podName:265a7e5e-277b-4ffa-ab70-f4d5b3e65f84 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:12.318602245 +0000 UTC m=+115.873758535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gsk84" (UID: "265a7e5e-277b-4ffa-ab70-f4d5b3e65f84") : secret "samples-operator-tls" not found
Sep 05 02:20:04.922926 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:04.922867    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:04.923358 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:04.922953    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:04.923358 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:04.923036    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:04.923358 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:04.923054    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:04.923358 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:04.923089    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm podName:b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b nodeName:}" failed. No retries permitted until 2026-09-05 02:20:08.923074871 +0000 UTC m=+112.478231162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdzmq" (UID: "b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:04.923358 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:04.923107    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls podName:b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b nodeName:}" failed. No retries permitted until 2026-09-05 02:20:08.923101365 +0000 UTC m=+112.478257652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdzmq" (UID: "b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:04.946706 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:04.946679    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lpd6d_2836c4ea-8e45-4bd3-9c96-ace613f40ed2/node-ca/0.log"
Sep 05 02:20:04.947045 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:04.947026    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:05.553123 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:05.553098    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5t47h_c7310203-e7ca-467e-945c-d4a9db60e8fd/serve-healthcheck-canary/0.log"
Sep 05 02:20:05.680285 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:05.680251    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:05.815726 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:05.815646    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:05.913684 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:05.913648    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:06.025784 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:06.025752    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:06.550558 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:06.550529    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-54qc4_12667c29-3724-4078-964d-07268713477a/kube-storage-version-migrator-operator/0.log"
Sep 05 02:20:06.931057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:06.931030    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:20:06.931231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:06.931102    2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:20:06.931455 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:06.931424    2557 scope.go:117] "RemoveContainer" containerID="7166725e19a1bf66c499665a273faff39fcbab135ea8cebe00c65159d16e2a25"
Sep 05 02:20:06.931619 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:06.931596    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-gtf6d_openshift-console-operator(83157c8a-294b-4add-aa27-3db843d0a9e5)\"" pod="openshift-console-operator/console-operator-85b454874b-gtf6d" podUID="83157c8a-294b-4add-aa27-3db843d0a9e5"
Sep 05 02:20:07.193260 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:07.193176    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:07.292876 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:07.292847    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:07.727582 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:07.727543    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:08.138282 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:08.138247    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:08.383701 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:08.383668    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:08.540326 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:08.540231    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:08.782037 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:08.782002    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:08.949473 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:08.949415    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:08.958428 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:08.958400    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:08.958509 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:08.958496    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:08.958560 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:08.958544    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:08.958618 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:08.958606    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls podName:b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b nodeName:}" failed. No retries permitted until 2026-09-05 02:20:16.958589089 +0000 UTC m=+120.513745376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdzmq" (UID: "b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:20:08.958661 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:08.958613    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:20:08.958661 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:08.958658    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm podName:b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b nodeName:}" failed. No retries permitted until 2026-09-05 02:20:16.958646672 +0000 UTC m=+120.513802961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdzmq" (UID: "b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b") : configmap "kube-rbac-proxy" not found
Sep 05 02:20:09.330560 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:09.330519    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:09.464318 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:09.464280    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:09.741507 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:09.741414    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:10.898825 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:10.898790    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:10.975311 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:10.975279    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:12.282245 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:12.282215    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:20:12.284499 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:12.284480    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls\") pod \"image-registry-6754f8dbcf-m2qqr\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") " pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:20:12.382997 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:12.382962    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gsk84\" (UID: \"265a7e5e-277b-4ffa-ab70-f4d5b3e65f84\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"
Sep 05 02:20:12.385696 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:12.385666    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/265a7e5e-277b-4ffa-ab70-f4d5b3e65f84-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gsk84\" (UID: \"265a7e5e-277b-4ffa-ab70-f4d5b3e65f84\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"
Sep 05 02:20:12.436447 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:12.436402    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:20:12.523833 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:12.523797    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"
Sep 05 02:20:12.563273 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:12.563174    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:12.635922 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:12.635878    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:12.713007 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:12.712981    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6754f8dbcf-m2qqr"]
Sep 05 02:20:12.715987 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:20:12.715962    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d1120c_2be6_4408_9862_61d45dc9625a.slice/crio-1f91ce98e859d6a5baf6450ffcaafc637ea76b0174ce3d7ff4b76aca6a4e38f2 WatchSource:0}: Error finding container 1f91ce98e859d6a5baf6450ffcaafc637ea76b0174ce3d7ff4b76aca6a4e38f2: Status 404 returned error can't find the container with id 1f91ce98e859d6a5baf6450ffcaafc637ea76b0174ce3d7ff4b76aca6a4e38f2
Sep 05 02:20:12.722957 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:12.722929    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:12.788816 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:12.788793    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84"]
Sep 05 02:20:13.473305 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:13.473264    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr" event={"ID":"b4d1120c-2be6-4408-9862-61d45dc9625a","Type":"ContainerStarted","Data":"f26e0cfa711a9c74d58687e3882bb68c069d8b83da3d8c6f87e5cc099b1770d0"}
Sep 05 02:20:13.473830 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:13.473341    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:20:13.473830 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:13.473365    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr" event={"ID":"b4d1120c-2be6-4408-9862-61d45dc9625a","Type":"ContainerStarted","Data":"1f91ce98e859d6a5baf6450ffcaafc637ea76b0174ce3d7ff4b76aca6a4e38f2"}
Sep 05 02:20:13.474576 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:13.474545    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84" event={"ID":"265a7e5e-277b-4ffa-ab70-f4d5b3e65f84","Type":"ContainerStarted","Data":"6795920b3e06ad409f54a0602a8378846966b6d9fde3a0847a529dc49d0d8ec7"}
Sep 05 02:20:13.493276 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:13.493216    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr" podStartSLOduration=17.493199326 podStartE2EDuration="17.493199326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:13.492081166 +0000 UTC m=+117.047237477" watchObservedRunningTime="2026-09-05 02:20:13.493199326 +0000 UTC m=+117.048355636"
Sep 05 02:20:14.737830 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:14.737757    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:14.838482 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:14.838450    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:15.482620 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:15.482581    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84" event={"ID":"265a7e5e-277b-4ffa-ab70-f4d5b3e65f84","Type":"ContainerStarted","Data":"20fa1f981938ab79a9b5de249bc67e90c159e3aa1be2cc7e3729c5872954bf8c"}
Sep 05 02:20:15.482814 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:15.482641    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84" event={"ID":"265a7e5e-277b-4ffa-ab70-f4d5b3e65f84","Type":"ContainerStarted","Data":"1367e71c19034da239c1ade23eed256c4b255568c5083aa40dde3fdc2a2bed0f"}
Sep 05 02:20:15.500183 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:15.500133    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gsk84" podStartSLOduration=17.852732498 podStartE2EDuration="19.500117711s" totalImagesPullingTime="1.647385213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:15.499737694 +0000 UTC m=+119.054894002" watchObservedRunningTime="2026-09-05 02:20:15.500117711 +0000 UTC m=+119.055274022"
Sep 05 02:20:15.522725 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:15.522690    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:15.747177 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:15.747097    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:16.120259 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:16.120226    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:16.324371 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:16.324325    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:16.710617 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:16.710583    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:17.022066 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:17.021979    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:17.022066 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:17.022042    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:17.022550 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:17.022500    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:17.024168 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:17.024145    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdzmq\" (UID: \"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b\") " pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:17.115178 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:17.115151    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mx6qs\""
Sep 05 02:20:17.122991 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:17.122968    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sdzmq"
Sep 05 02:20:17.406464 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:17.406386    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sdzmq"]
Sep 05 02:20:17.411654 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:20:17.411626    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ffd5ff_67dd_4937_bdbe_5b5f1e45ca1b.slice/crio-d4e09cd4c4a37424677924b8917d4b93def8da8a9df7ee2881c3edb755811085 WatchSource:0}: Error finding container d4e09cd4c4a37424677924b8917d4b93def8da8a9df7ee2881c3edb755811085: Status 404 returned error can't find the container with id d4e09cd4c4a37424677924b8917d4b93def8da8a9df7ee2881c3edb755811085
Sep 05 02:20:17.494861 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:17.494832    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdzmq" event={"ID":"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b","Type":"ContainerStarted","Data":"62796a7126c89bb4da08ccc5715b2fce86757b052c1a2933dd87ab16302faf98"}
Sep 05 02:20:17.494982 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:17.494887    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdzmq" event={"ID":"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b","Type":"ContainerStarted","Data":"d4e09cd4c4a37424677924b8917d4b93def8da8a9df7ee2881c3edb755811085"}
Sep 05 02:20:17.985119 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:17.985091    2557 scope.go:117] "RemoveContainer" containerID="7166725e19a1bf66c499665a273faff39fcbab135ea8cebe00c65159d16e2a25"
Sep 05 02:20:18.499083 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:18.499045    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdzmq" event={"ID":"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b","Type":"ContainerStarted","Data":"a4b2539efb789b632ebff72a9dac0af7c1d180368a0972c537956714f398cdbd"}
Sep 05 02:20:18.500463 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:18.500432    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:20:18.500834 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:18.500818    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/1.log"
Sep 05 02:20:18.500872 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:18.500850    2557 generic.go:356] "Generic (PLEG): container finished" podID="83157c8a-294b-4add-aa27-3db843d0a9e5" containerID="c67474c82daf8fc85fbbcca91805c6c67ac1cdfdd89df43e937c5681c1ec3d67" exitCode=255
Sep 05 02:20:18.500907 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:18.500897    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-gtf6d" event={"ID":"83157c8a-294b-4add-aa27-3db843d0a9e5","Type":"ContainerDied","Data":"c67474c82daf8fc85fbbcca91805c6c67ac1cdfdd89df43e937c5681c1ec3d67"}
Sep 05 02:20:18.500953 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:18.500939    2557 scope.go:117] "RemoveContainer" containerID="7166725e19a1bf66c499665a273faff39fcbab135ea8cebe00c65159d16e2a25"
Sep 05 02:20:18.501240 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:18.501207    2557 scope.go:117] "RemoveContainer" containerID="c67474c82daf8fc85fbbcca91805c6c67ac1cdfdd89df43e937c5681c1ec3d67"
Sep 05 02:20:18.501396 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:18.501379    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-gtf6d_openshift-console-operator(83157c8a-294b-4add-aa27-3db843d0a9e5)\"" pod="openshift-console-operator/console-operator-85b454874b-gtf6d" podUID="83157c8a-294b-4add-aa27-3db843d0a9e5"
Sep 05 02:20:19.506368 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:19.506340    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:20:19.867026 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:19.866979    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:19.929762 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:19.929731    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:20.511796 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:20.511759    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdzmq" event={"ID":"b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b","Type":"ContainerStarted","Data":"bef36a65cd9a7d3ed922c91adab7fbee67cf8ecc5917d7d7cd16e72f4211ece2"}
Sep 05 02:20:20.529728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:20.529678    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sdzmq" podStartSLOduration=17.139661085 podStartE2EDuration="19.529660881s" totalImagesPullingTime="2.389999796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:20:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:20.529615473 +0000 UTC m=+124.084771782" watchObservedRunningTime="2026-09-05 02:20:20.529660881 +0000 UTC m=+124.084817190"
Sep 05 02:20:24.372024 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.371991    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586469f8d6-lp5kw"]
Sep 05 02:20:24.375090 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.375057    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586469f8d6-lp5kw"
Sep 05 02:20:24.379057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.379031    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 05 02:20:24.379940 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.379916    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 05 02:20:24.379940 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.379924    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wtw85\""
Sep 05 02:20:24.380130 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.379970    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 05 02:20:24.380130 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.379974    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 05 02:20:24.386671 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.386652    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"]
Sep 05 02:20:24.389433 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.389418    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586469f8d6-lp5kw"]
Sep 05 02:20:24.389674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.389534    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"
Sep 05 02:20:24.391509 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.391489    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 05 02:20:24.411428 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.411407    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"]
Sep 05 02:20:24.483408 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.480238    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msh7\" (UniqueName: \"kubernetes.io/projected/35766fad-84d3-4486-bea3-1bd4f2691596-kube-api-access-9msh7\") pod \"managed-serviceaccount-addon-agent-586469f8d6-lp5kw\" (UID: \"35766fad-84d3-4486-bea3-1bd4f2691596\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586469f8d6-lp5kw"
Sep 05 02:20:24.483408 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.480332    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35766fad-84d3-4486-bea3-1bd4f2691596-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-586469f8d6-lp5kw\" (UID: \"35766fad-84d3-4486-bea3-1bd4f2691596\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586469f8d6-lp5kw"
Sep 05 02:20:24.483408 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.480498    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c083da31-d5c9-4297-aca3-5a7680737998-tmp\") pod \"klusterlet-addon-workmgr-6d44f6989f-jnzzq\" (UID: \"c083da31-d5c9-4297-aca3-5a7680737998\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"
Sep 05 02:20:24.483408 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.480570    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqt8r\" (UniqueName: \"kubernetes.io/projected/c083da31-d5c9-4297-aca3-5a7680737998-kube-api-access-nqt8r\") pod \"klusterlet-addon-workmgr-6d44f6989f-jnzzq\" (UID: \"c083da31-d5c9-4297-aca3-5a7680737998\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"
Sep 05 02:20:24.483408 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.480664    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c083da31-d5c9-4297-aca3-5a7680737998-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d44f6989f-jnzzq\" (UID: \"c083da31-d5c9-4297-aca3-5a7680737998\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"
Sep 05 02:20:24.483408 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.481225    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6754f8dbcf-m2qqr"]
Sep 05 02:20:24.525284 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.525231    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56897c4b7f-7m4jc"]
Sep 05 02:20:24.528221 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.528205    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.547849 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.547819    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56897c4b7f-7m4jc"]
Sep 05 02:20:24.564316 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.564279    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:24.581711 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.581684    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msh7\" (UniqueName: \"kubernetes.io/projected/35766fad-84d3-4486-bea3-1bd4f2691596-kube-api-access-9msh7\") pod \"managed-serviceaccount-addon-agent-586469f8d6-lp5kw\" (UID: \"35766fad-84d3-4486-bea3-1bd4f2691596\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586469f8d6-lp5kw"
Sep 05 02:20:24.581827 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.581724    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35766fad-84d3-4486-bea3-1bd4f2691596-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-586469f8d6-lp5kw\" (UID: \"35766fad-84d3-4486-bea3-1bd4f2691596\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586469f8d6-lp5kw"
Sep 05 02:20:24.581827 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.581756    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c083da31-d5c9-4297-aca3-5a7680737998-tmp\") pod \"klusterlet-addon-workmgr-6d44f6989f-jnzzq\" (UID: \"c083da31-d5c9-4297-aca3-5a7680737998\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"
Sep 05 02:20:24.581827 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.581809    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqt8r\" (UniqueName: \"kubernetes.io/projected/c083da31-d5c9-4297-aca3-5a7680737998-kube-api-access-nqt8r\") pod \"klusterlet-addon-workmgr-6d44f6989f-jnzzq\" (UID: \"c083da31-d5c9-4297-aca3-5a7680737998\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"
Sep 05 02:20:24.581950 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.581865    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c083da31-d5c9-4297-aca3-5a7680737998-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d44f6989f-jnzzq\" (UID: \"c083da31-d5c9-4297-aca3-5a7680737998\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"
Sep 05 02:20:24.582146 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.582125    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c083da31-d5c9-4297-aca3-5a7680737998-tmp\") pod \"klusterlet-addon-workmgr-6d44f6989f-jnzzq\" (UID: \"c083da31-d5c9-4297-aca3-5a7680737998\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"
Sep 05 02:20:24.584210 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.584189    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c083da31-d5c9-4297-aca3-5a7680737998-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d44f6989f-jnzzq\" (UID: \"c083da31-d5c9-4297-aca3-5a7680737998\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"
Sep 05 02:20:24.584210 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.584205    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35766fad-84d3-4486-bea3-1bd4f2691596-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-586469f8d6-lp5kw\" (UID: \"35766fad-84d3-4486-bea3-1bd4f2691596\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586469f8d6-lp5kw"
Sep 05 02:20:24.590744 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.590718    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqt8r\" (UniqueName: \"kubernetes.io/projected/c083da31-d5c9-4297-aca3-5a7680737998-kube-api-access-nqt8r\") pod \"klusterlet-addon-workmgr-6d44f6989f-jnzzq\" (UID: \"c083da31-d5c9-4297-aca3-5a7680737998\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"
Sep 05 02:20:24.590832 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.590810    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msh7\" (UniqueName: \"kubernetes.io/projected/35766fad-84d3-4486-bea3-1bd4f2691596-kube-api-access-9msh7\") pod \"managed-serviceaccount-addon-agent-586469f8d6-lp5kw\" (UID: \"35766fad-84d3-4486-bea3-1bd4f2691596\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586469f8d6-lp5kw"
Sep 05 02:20:24.660604 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.660539    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:24.682548 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.682525    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-trusted-ca\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.682634 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.682553    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-image-registry-private-configuration\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.682634 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.682571    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-registry-certificates\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.682634 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.682590    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxcvq\" (UniqueName: \"kubernetes.io/projected/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-kube-api-access-mxcvq\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.682634 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.682624    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-ca-trust-extracted\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.682766 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.682665    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-registry-tls\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.682766 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.682717    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-bound-sa-token\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.682827 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.682778    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-installation-pull-secrets\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.694059 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.694042    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586469f8d6-lp5kw"
Sep 05 02:20:24.701958 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.701941    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"
Sep 05 02:20:24.768917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.768882    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:24.783944 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.783920    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-installation-pull-secrets\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.784063 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.783963    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-trusted-ca\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.784063 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.783986    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-image-registry-private-configuration\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.784063 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.784011    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-registry-certificates\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.784063 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.784042    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxcvq\" (UniqueName: \"kubernetes.io/projected/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-kube-api-access-mxcvq\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.784257 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.784091    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-ca-trust-extracted\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.784257 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.784122    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-registry-tls\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.784257 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.784162    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-bound-sa-token\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.784694 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.784655    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-ca-trust-extracted\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.784987 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.784969    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-trusted-ca\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.785754 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.785700    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-registry-certificates\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.786596 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.786578    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-registry-tls\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.786845 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.786831    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-installation-pull-secrets\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.787987 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.787961    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-image-registry-private-configuration\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.805506 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.805484    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-bound-sa-token\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.811284 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.811262    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxcvq\" (UniqueName: \"kubernetes.io/projected/c3aa1a59-6ce1-4ae7-93d7-d680837d353d-kube-api-access-mxcvq\") pod \"image-registry-56897c4b7f-7m4jc\" (UID: \"c3aa1a59-6ce1-4ae7-93d7-d680837d353d\") " pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.836150 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.836129    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:24.971956 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.971890    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586469f8d6-lp5kw"]
Sep 05 02:20:24.975006 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:20:24.974967    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35766fad_84d3_4486_bea3_1bd4f2691596.slice/crio-ddbdc3adeecab2fd793136104717c7318ee0b5368d759be2bffe452ade2cdeb3 WatchSource:0}: Error finding container ddbdc3adeecab2fd793136104717c7318ee0b5368d759be2bffe452ade2cdeb3: Status 404 returned error can't find the container with id ddbdc3adeecab2fd793136104717c7318ee0b5368d759be2bffe452ade2cdeb3
Sep 05 02:20:24.976501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:24.976470    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"]
Sep 05 02:20:24.978832 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:20:24.978812    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc083da31_d5c9_4297_aca3_5a7680737998.slice/crio-f9d13aafb2accaac8ed399b9490280d3c1435f5c3388f0787057045b419e1fdf WatchSource:0}: Error finding container f9d13aafb2accaac8ed399b9490280d3c1435f5c3388f0787057045b419e1fdf: Status 404 returned error can't find the container with id f9d13aafb2accaac8ed399b9490280d3c1435f5c3388f0787057045b419e1fdf
Sep 05 02:20:25.106430 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:25.106400    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56897c4b7f-7m4jc"]
Sep 05 02:20:25.109070 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:20:25.109046    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3aa1a59_6ce1_4ae7_93d7_d680837d353d.slice/crio-b7caaf7387df6e95b83981447637344d54eecd0152e2d63849c3a883c3c89900 WatchSource:0}: Error finding container b7caaf7387df6e95b83981447637344d54eecd0152e2d63849c3a883c3c89900: Status 404 returned error can't find the container with id b7caaf7387df6e95b83981447637344d54eecd0152e2d63849c3a883c3c89900
Sep 05 02:20:25.529316 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:25.529280    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq" event={"ID":"c083da31-d5c9-4297-aca3-5a7680737998","Type":"ContainerStarted","Data":"f9d13aafb2accaac8ed399b9490280d3c1435f5c3388f0787057045b419e1fdf"}
Sep 05 02:20:25.530862 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:25.530830    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc" event={"ID":"c3aa1a59-6ce1-4ae7-93d7-d680837d353d","Type":"ContainerStarted","Data":"b30e32d2b3491f81e4c4cb2ccc999d6106e6f6600d367cb7d99c42dadea5c3a1"}
Sep 05 02:20:25.530980 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:25.530868    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc" event={"ID":"c3aa1a59-6ce1-4ae7-93d7-d680837d353d","Type":"ContainerStarted","Data":"b7caaf7387df6e95b83981447637344d54eecd0152e2d63849c3a883c3c89900"}
Sep 05 02:20:25.530980 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:25.530954    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:20:25.532095 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:25.532067    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586469f8d6-lp5kw" event={"ID":"35766fad-84d3-4486-bea3-1bd4f2691596","Type":"ContainerStarted","Data":"ddbdc3adeecab2fd793136104717c7318ee0b5368d759be2bffe452ade2cdeb3"}
Sep 05 02:20:25.550679 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:25.550638    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc" podStartSLOduration=1.5506251770000001 podStartE2EDuration="1.550625177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:25.549790855 +0000 UTC m=+129.104947175" watchObservedRunningTime="2026-09-05 02:20:25.550625177 +0000 UTC m=+129.105781485"
Sep 05 02:20:26.701917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:26.701654    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs\") pod \"network-metrics-daemon-2c797\" (UID: \"b933b52f-75e9-478b-a95b-5cb77d236467\") " pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:20:26.704608 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:26.704575    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b933b52f-75e9-478b-a95b-5cb77d236467-metrics-certs\") pod \"network-metrics-daemon-2c797\" (UID: \"b933b52f-75e9-478b-a95b-5cb77d236467\") " pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:20:26.897354 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:26.897130    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zqlb9\""
Sep 05 02:20:26.904928 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:26.904705    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2c797"
Sep 05 02:20:26.930707 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:26.930672    2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:20:26.930838 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:26.930732    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:20:26.931291 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:26.931271    2557 scope.go:117] "RemoveContainer" containerID="c67474c82daf8fc85fbbcca91805c6c67ac1cdfdd89df43e937c5681c1ec3d67"
Sep 05 02:20:26.931579 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:26.931523    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-gtf6d_openshift-console-operator(83157c8a-294b-4add-aa27-3db843d0a9e5)\"" pod="openshift-console-operator/console-operator-85b454874b-gtf6d" podUID="83157c8a-294b-4add-aa27-3db843d0a9e5"
Sep 05 02:20:27.051793 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:27.051712    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:27.122291 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:27.122208    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:29.552054 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:29.552018    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq" event={"ID":"c083da31-d5c9-4297-aca3-5a7680737998","Type":"ContainerStarted","Data":"ffdea933f3b77ddd5790a13598c6ffdcf3683d4c8da3bb56decd981dfacc4a65"}
Sep 05 02:20:29.552546 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:29.552263    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"
Sep 05 02:20:29.554087 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:29.553903    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq" podUID="c083da31-d5c9-4297-aca3-5a7680737998" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.16:8000/readyz\": dial tcp 10.133.0.16:8000: connect: connection refused"
Sep 05 02:20:29.554087 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:29.553999    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586469f8d6-lp5kw" event={"ID":"35766fad-84d3-4486-bea3-1bd4f2691596","Type":"ContainerStarted","Data":"7f8cbfec33869858322cc034a45b556d181e1380a3cf586969712a4e374d5d23"}
Sep 05 02:20:29.567318 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:29.567233    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq" podStartSLOduration=1.070376196 podStartE2EDuration="5.567202881s" totalImagesPullingTime="4.496826685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:29.566805913 +0000 UTC m=+133.121962223" watchObservedRunningTime="2026-09-05 02:20:29.567202881 +0000 UTC m=+133.122359191"
Sep 05 02:20:29.582611 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:29.582515    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586469f8d6-lp5kw" podStartSLOduration=1.096359922 podStartE2EDuration="5.58249747s" totalImagesPullingTime="4.486137548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:29.581264798 +0000 UTC m=+133.136421107" watchObservedRunningTime="2026-09-05 02:20:29.58249747 +0000 UTC m=+133.137653778"
Sep 05 02:20:29.724580 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:29.724503    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2c797"]
Sep 05 02:20:29.727142 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:20:29.727118    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb933b52f_75e9_478b_a95b_5cb77d236467.slice/crio-d3572eb264813ab9a673d4227f20a3b9ec8ebb09a0500ec46ce96bcc8875156d WatchSource:0}: Error finding container d3572eb264813ab9a673d4227f20a3b9ec8ebb09a0500ec46ce96bcc8875156d: Status 404 returned error can't find the container with id d3572eb264813ab9a673d4227f20a3b9ec8ebb09a0500ec46ce96bcc8875156d
Sep 05 02:20:30.558417 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:30.558382    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2c797" event={"ID":"b933b52f-75e9-478b-a95b-5cb77d236467","Type":"ContainerStarted","Data":"d3572eb264813ab9a673d4227f20a3b9ec8ebb09a0500ec46ce96bcc8875156d"}
Sep 05 02:20:30.559614 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:30.559594    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d44f6989f-jnzzq"
Sep 05 02:20:30.852801 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:30.852768    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:30.949698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:30.949647    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:31.034005 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:31.033977    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:31.169750 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:31.169722    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:31.565005 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:31.564924    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2c797" event={"ID":"b933b52f-75e9-478b-a95b-5cb77d236467","Type":"ContainerStarted","Data":"b917837f50f093655bc3ca89b40897c0ebe2f4437766f511650dfe01d6e1bac8"}
Sep 05 02:20:31.565363 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:31.565006    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2c797" event={"ID":"b933b52f-75e9-478b-a95b-5cb77d236467","Type":"ContainerStarted","Data":"94cec8b17c769fb2acdb8825af49f8ab6a92263eccd837d8197d350fef62e8e4"}
Sep 05 02:20:31.582346 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:31.582297    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2c797" podStartSLOduration=133.401061417 podStartE2EDuration="2m14.582283534s" totalImagesPullingTime="1.181222117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:31.580910822 +0000 UTC m=+135.136067146" watchObservedRunningTime="2026-09-05 02:20:31.582283534 +0000 UTC m=+135.137439844"
Sep 05 02:20:31.692147 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:31.692110    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:31.765392 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:31.765362    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:32.608194 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:32.608161    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:32.708743 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:32.708715    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:34.045744 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:34.045710    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:34.112885 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:34.112855    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:34.487032 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:34.487000    2557 patch_prober.go:28] interesting pod/image-registry-6754f8dbcf-m2qqr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 05 02:20:34.487233 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:34.487057    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr" podUID="b4d1120c-2be6-4408-9862-61d45dc9625a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:20:34.738660 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:34.738573    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:36.753620 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:36.753586    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:38.184790 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.184751    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:38.369206 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.369170    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:38.457246 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.457172    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s6dmz"]
Sep 05 02:20:38.459702 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.459686    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.461890 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.461851    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 05 02:20:38.461890 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.461858    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 05 02:20:38.462160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.462146    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 05 02:20:38.462719 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.462697    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ld2zj\""
Sep 05 02:20:38.462834 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.462762    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 05 02:20:38.462834 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.462808    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 05 02:20:38.462951 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.462873    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 05 02:20:38.470597 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.470576    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:38.501634 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.501610    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-textfile\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.501760 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.501643    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.501760 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.501663    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-metrics-client-ca\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.501760 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.501729    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-sys\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.501760 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.501754    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-tls\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.501917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.501769    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-root\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.501917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.501794    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtgc7\" (UniqueName: \"kubernetes.io/projected/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-kube-api-access-wtgc7\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.501917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.501870    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.502027 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.501915    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-wtmp\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.602734 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.602709    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.602894 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.602751    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-wtmp\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.602894 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.602779    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-textfile\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.602894 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.602801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.602894 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.602820    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-metrics-client-ca\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.602894 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.602846    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-sys\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.602894 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.602867    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-tls\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.602894 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.602883    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-root\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.603235 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.602904    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-wtmp\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.603235 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.602927    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtgc7\" (UniqueName: \"kubernetes.io/projected/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-kube-api-access-wtgc7\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.603235 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.602964    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-sys\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.603235 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:38.602982    2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 05 02:20:38.603235 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:38.603042    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-tls podName:d010c7b3-2a2d-4aef-b279-2948b7f0b0f9 nodeName:}" failed. No retries permitted until 2026-09-05 02:20:39.103027734 +0000 UTC m=+142.658184020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-tls") pod "node-exporter-s6dmz" (UID: "d010c7b3-2a2d-4aef-b279-2948b7f0b0f9") : secret "node-exporter-tls" not found
Sep 05 02:20:38.603235 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.603081    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-root\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.603574 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.603304    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-textfile\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.603574 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.603465    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.603574 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.603562    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-metrics-client-ca\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.604983 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.604966    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.612375 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.612344    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtgc7\" (UniqueName: \"kubernetes.io/projected/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-kube-api-access-wtgc7\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:38.771892 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.771809    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:38.869951 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.869920    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:38.970700 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:38.970672    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:39.106291 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:39.106265    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-tls\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:39.108491 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:39.108466    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d010c7b3-2a2d-4aef-b279-2948b7f0b0f9-node-exporter-tls\") pod \"node-exporter-s6dmz\" (UID: \"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9\") " pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:39.369362 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:39.369291    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s6dmz"
Sep 05 02:20:39.379244 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:20:39.379210    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd010c7b3_2a2d_4aef_b279_2948b7f0b0f9.slice/crio-02b7b2dc926199b18bbfc55e28e65360de59d4ff6a25ecd5db8106683d733e53 WatchSource:0}: Error finding container 02b7b2dc926199b18bbfc55e28e65360de59d4ff6a25ecd5db8106683d733e53: Status 404 returned error can't find the container with id 02b7b2dc926199b18bbfc55e28e65360de59d4ff6a25ecd5db8106683d733e53
Sep 05 02:20:39.413508 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:39.413482    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:39.490308 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:39.490276    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:39.592591 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:39.592560    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6dmz" event={"ID":"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9","Type":"ContainerStarted","Data":"02b7b2dc926199b18bbfc55e28e65360de59d4ff6a25ecd5db8106683d733e53"}
Sep 05 02:20:39.614620 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:39.614592    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:39.734261 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:39.734183    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:40.993323 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:40.993302    2557 scope.go:117] "RemoveContainer" containerID="c67474c82daf8fc85fbbcca91805c6c67ac1cdfdd89df43e937c5681c1ec3d67"
Sep 05 02:20:41.347897 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:41.347862    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:41.406355 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:41.406319    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:41.492331 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:41.492296    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:41.556408 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:41.556373    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:41.602937 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:41.602841    2557 generic.go:356] "Generic (PLEG): container finished" podID="d010c7b3-2a2d-4aef-b279-2948b7f0b0f9" containerID="0a537cbb57836a3a2cbf52776ba3a55c0d406c51ef5441825116e8aff9e37bc7" exitCode=0
Sep 05 02:20:41.602937 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:41.602890    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6dmz" event={"ID":"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9","Type":"ContainerDied","Data":"0a537cbb57836a3a2cbf52776ba3a55c0d406c51ef5441825116e8aff9e37bc7"}
Sep 05 02:20:41.604975 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:41.604958    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:20:41.605075 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:41.604999    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-gtf6d" event={"ID":"83157c8a-294b-4add-aa27-3db843d0a9e5","Type":"ContainerStarted","Data":"0ac666ec4bcd5387840b77d8ac3216380bad0d82a620273c0552575567c0ad2b"}
Sep 05 02:20:41.605283 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:41.605265    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:20:41.610532 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:41.610512    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-gtf6d"
Sep 05 02:20:41.647995 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:41.647943    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-gtf6d" podStartSLOduration=42.738992469 podStartE2EDuration="45.647923733s" totalImagesPullingTime="2.908931264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:41.646616314 +0000 UTC m=+145.201772622" watchObservedRunningTime="2026-09-05 02:20:41.647923733 +0000 UTC m=+145.203080055"
Sep 05 02:20:41.782333 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:41.782297    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:41.877844 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:41.877753    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:42.058024 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:42.057980    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:42.220158 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:42.220072    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:42.511765 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:42.511676    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:42.610632 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:42.610598    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6dmz" event={"ID":"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9","Type":"ContainerStarted","Data":"a8d5ca1650c6b8da597faf405bbfac2a15c18dcae217ffb9640d85c5dee70cda"}
Sep 05 02:20:42.610781 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:42.610654    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6dmz" event={"ID":"d010c7b3-2a2d-4aef-b279-2948b7f0b0f9","Type":"ContainerStarted","Data":"edd2c83012dc9777392c9541c5d7483c63d5d211cba167b7d5d76177bc929aa0"}
Sep 05 02:20:42.612660 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:42.612636    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:42.633738 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:42.633696    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s6dmz" podStartSLOduration=3.198471963 podStartE2EDuration="4.633683228s" totalImagesPullingTime="1.375063703s" totalInitContainerRuntime="60.147562ms" isStatefulPod=true podCreationTimestamp="2026-09-05 02:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:42.63342509 +0000 UTC m=+146.188581398" watchObservedRunningTime="2026-09-05 02:20:42.633683228 +0000 UTC m=+146.188839537"
Sep 05 02:20:43.146896 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:43.146865    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:43.241341 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:43.241306    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:43.304840 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:43.304800    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:43.637971 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:43.637941    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:43.828032 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:43.828001    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:43.953587 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:43.953511    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:44.485756 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:44.485724    2557 patch_prober.go:28] interesting pod/image-registry-6754f8dbcf-m2qqr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 05 02:20:44.486226 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:44.485779    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr" podUID="b4d1120c-2be6-4408-9862-61d45dc9625a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:20:44.490642 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:44.490612    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:44.591829 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:44.591793    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:44.693872 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:44.693839    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:44.840776 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:44.840741    2557 patch_prober.go:28] interesting pod/image-registry-56897c4b7f-7m4jc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 05 02:20:44.840940 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:44.840803    2557 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc" podUID="c3aa1a59-6ce1-4ae7-93d7-d680837d353d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:20:46.541453 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:46.541399    2557 patch_prober.go:28] interesting pod/image-registry-56897c4b7f-7m4jc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 05 02:20:46.541815 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:46.541484    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc" podUID="c3aa1a59-6ce1-4ae7-93d7-d680837d353d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:20:49.501677 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.501634    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr" podUID="b4d1120c-2be6-4408-9862-61d45dc9625a" containerName="registry" containerID="cri-o://f26e0cfa711a9c74d58687e3882bb68c069d8b83da3d8c6f87e5cc099b1770d0" gracePeriod=30
Sep 05 02:20:49.640120 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.640076    2557 generic.go:356] "Generic (PLEG): container finished" podID="b4d1120c-2be6-4408-9862-61d45dc9625a" containerID="f26e0cfa711a9c74d58687e3882bb68c069d8b83da3d8c6f87e5cc099b1770d0" exitCode=0
Sep 05 02:20:49.640263 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.640149    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr" event={"ID":"b4d1120c-2be6-4408-9862-61d45dc9625a","Type":"ContainerDied","Data":"f26e0cfa711a9c74d58687e3882bb68c069d8b83da3d8c6f87e5cc099b1770d0"}
Sep 05 02:20:49.738205 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.738183    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:20:49.791728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.791659    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-bound-sa-token\") pod \"b4d1120c-2be6-4408-9862-61d45dc9625a\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") "
Sep 05 02:20:49.791728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.791693    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4d1120c-2be6-4408-9862-61d45dc9625a-installation-pull-secrets\") pod \"b4d1120c-2be6-4408-9862-61d45dc9625a\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") "
Sep 05 02:20:49.791917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.791735    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls\") pod \"b4d1120c-2be6-4408-9862-61d45dc9625a\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") "
Sep 05 02:20:49.791917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.791773    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4d1120c-2be6-4408-9862-61d45dc9625a-ca-trust-extracted\") pod \"b4d1120c-2be6-4408-9862-61d45dc9625a\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") "
Sep 05 02:20:49.791917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.791804    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-certificates\") pod \"b4d1120c-2be6-4408-9862-61d45dc9625a\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") "
Sep 05 02:20:49.791917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.791825    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf8jr\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-kube-api-access-jf8jr\") pod \"b4d1120c-2be6-4408-9862-61d45dc9625a\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") "
Sep 05 02:20:49.791917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.791846    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4d1120c-2be6-4408-9862-61d45dc9625a-image-registry-private-configuration\") pod \"b4d1120c-2be6-4408-9862-61d45dc9625a\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") "
Sep 05 02:20:49.791917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.791901    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4d1120c-2be6-4408-9862-61d45dc9625a-trusted-ca\") pod \"b4d1120c-2be6-4408-9862-61d45dc9625a\" (UID: \"b4d1120c-2be6-4408-9862-61d45dc9625a\") "
Sep 05 02:20:49.792404 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.792369    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b4d1120c-2be6-4408-9862-61d45dc9625a" (UID: "b4d1120c-2be6-4408-9862-61d45dc9625a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:49.792575 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.792423    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d1120c-2be6-4408-9862-61d45dc9625a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b4d1120c-2be6-4408-9862-61d45dc9625a" (UID: "b4d1120c-2be6-4408-9862-61d45dc9625a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:49.800415 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.800375    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b4d1120c-2be6-4408-9862-61d45dc9625a" (UID: "b4d1120c-2be6-4408-9862-61d45dc9625a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:49.800544 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.800454    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d1120c-2be6-4408-9862-61d45dc9625a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b4d1120c-2be6-4408-9862-61d45dc9625a" (UID: "b4d1120c-2be6-4408-9862-61d45dc9625a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:49.800610 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.800559    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-kube-api-access-jf8jr" (OuterVolumeSpecName: "kube-api-access-jf8jr") pod "b4d1120c-2be6-4408-9862-61d45dc9625a" (UID: "b4d1120c-2be6-4408-9862-61d45dc9625a"). InnerVolumeSpecName "kube-api-access-jf8jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:49.800676 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.800648    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b4d1120c-2be6-4408-9862-61d45dc9625a" (UID: "b4d1120c-2be6-4408-9862-61d45dc9625a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:49.801092 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.801068    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d1120c-2be6-4408-9862-61d45dc9625a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b4d1120c-2be6-4408-9862-61d45dc9625a" (UID: "b4d1120c-2be6-4408-9862-61d45dc9625a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:49.802192 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.802163    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4d1120c-2be6-4408-9862-61d45dc9625a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b4d1120c-2be6-4408-9862-61d45dc9625a" (UID: "b4d1120c-2be6-4408-9862-61d45dc9625a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:20:49.892597 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.892566    2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-tls\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:20:49.892597 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.892592    2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4d1120c-2be6-4408-9862-61d45dc9625a-ca-trust-extracted\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:20:49.892597 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.892602    2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4d1120c-2be6-4408-9862-61d45dc9625a-registry-certificates\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:20:49.892786 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.892611    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf8jr\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-kube-api-access-jf8jr\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:20:49.892786 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.892621    2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4d1120c-2be6-4408-9862-61d45dc9625a-image-registry-private-configuration\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:20:49.892786 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.892631    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4d1120c-2be6-4408-9862-61d45dc9625a-trusted-ca\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:20:49.892786 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.892639    2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4d1120c-2be6-4408-9862-61d45dc9625a-bound-sa-token\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:20:49.892786 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:49.892647    2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4d1120c-2be6-4408-9862-61d45dc9625a-installation-pull-secrets\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:20:50.644302 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:50.644265    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr" event={"ID":"b4d1120c-2be6-4408-9862-61d45dc9625a","Type":"ContainerDied","Data":"1f91ce98e859d6a5baf6450ffcaafc637ea76b0174ce3d7ff4b76aca6a4e38f2"}
Sep 05 02:20:50.644751 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:50.644325    2557 scope.go:117] "RemoveContainer" containerID="f26e0cfa711a9c74d58687e3882bb68c069d8b83da3d8c6f87e5cc099b1770d0"
Sep 05 02:20:50.644751 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:50.644274    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6754f8dbcf-m2qqr"
Sep 05 02:20:50.664955 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:50.664930    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6754f8dbcf-m2qqr"]
Sep 05 02:20:50.667408 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:50.667382    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6754f8dbcf-m2qqr"]
Sep 05 02:20:50.989021 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:50.988944    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4d1120c-2be6-4408-9862-61d45dc9625a" path="/var/lib/kubelet/pods/b4d1120c-2be6-4408-9862-61d45dc9625a/volumes"
Sep 05 02:20:51.668680 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.668648    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55f695f754-tr5cj"]
Sep 05 02:20:51.669036 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.668889    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4d1120c-2be6-4408-9862-61d45dc9625a" containerName="registry"
Sep 05 02:20:51.669036 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.668900    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4d1120c-2be6-4408-9862-61d45dc9625a" containerName="registry"
Sep 05 02:20:51.669036 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.668949    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4d1120c-2be6-4408-9862-61d45dc9625a" containerName="registry"
Sep 05 02:20:51.673503 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.673482    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.676169 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.676142    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 05 02:20:51.677012 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.676995    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 05 02:20:51.677110 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.676997    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 05 02:20:51.677394 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.677374    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 05 02:20:51.677507 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.677394    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m5q29\""
Sep 05 02:20:51.677507 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.677469    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 05 02:20:51.677507 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.677484    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 05 02:20:51.678072 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.678056    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 05 02:20:51.691164 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.691139    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f695f754-tr5cj"]
Sep 05 02:20:51.707259 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.707227    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-oauth-serving-cert\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.707372 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.707271    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ws78\" (UniqueName: \"kubernetes.io/projected/35134ac3-0010-441c-9adc-654aa2ac871a-kube-api-access-9ws78\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.707372 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.707315    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-console-config\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.707372 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.707335    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-service-ca\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.707372 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.707360    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35134ac3-0010-441c-9adc-654aa2ac871a-console-oauth-config\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.707548 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.707400    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35134ac3-0010-441c-9adc-654aa2ac871a-console-serving-cert\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.808422 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.808386    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35134ac3-0010-441c-9adc-654aa2ac871a-console-serving-cert\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.808587 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.808462    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-oauth-serving-cert\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.808587 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.808499    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ws78\" (UniqueName: \"kubernetes.io/projected/35134ac3-0010-441c-9adc-654aa2ac871a-kube-api-access-9ws78\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.808587 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.808550    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-console-config\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.808587 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.808575    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-service-ca\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.808813 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.808598    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35134ac3-0010-441c-9adc-654aa2ac871a-console-oauth-config\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.809210 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.809181    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-oauth-serving-cert\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.809321 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.809278    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-service-ca\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.809321 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.809287    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-console-config\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.810980 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.810942    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35134ac3-0010-441c-9adc-654aa2ac871a-console-serving-cert\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.811091 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.810991    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35134ac3-0010-441c-9adc-654aa2ac871a-console-oauth-config\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.816678 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.816656    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ws78\" (UniqueName: \"kubernetes.io/projected/35134ac3-0010-441c-9adc-654aa2ac871a-kube-api-access-9ws78\") pod \"console-55f695f754-tr5cj\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") " pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:51.983279 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:51.983192    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:20:52.271914 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:52.271833    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f695f754-tr5cj"]
Sep 05 02:20:52.274903 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:20:52.274872    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35134ac3_0010_441c_9adc_654aa2ac871a.slice/crio-38972e0592f247c5c46ab9ef5ad3f613008b5fdc8262584710f414da20f25e93 WatchSource:0}: Error finding container 38972e0592f247c5c46ab9ef5ad3f613008b5fdc8262584710f414da20f25e93: Status 404 returned error can't find the container with id 38972e0592f247c5c46ab9ef5ad3f613008b5fdc8262584710f414da20f25e93
Sep 05 02:20:52.652713 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:52.652676    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f695f754-tr5cj" event={"ID":"35134ac3-0010-441c-9adc-654aa2ac871a","Type":"ContainerStarted","Data":"38972e0592f247c5c46ab9ef5ad3f613008b5fdc8262584710f414da20f25e93"}
Sep 05 02:20:54.841419 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:54.841379    2557 patch_prober.go:28] interesting pod/image-registry-56897c4b7f-7m4jc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 05 02:20:54.841840 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:54.841466    2557 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc" podUID="c3aa1a59-6ce1-4ae7-93d7-d680837d353d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:20:55.292041 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:20:55.292012    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qs48v" podUID="19ece4dc-ca43-42a0-bf9c-64ab7e233025"
Sep 05 02:20:55.645817 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:55.645781    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:55.665809 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:55.665780    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qs48v"
Sep 05 02:20:55.665956 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:55.665807    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f695f754-tr5cj" event={"ID":"35134ac3-0010-441c-9adc-654aa2ac871a","Type":"ContainerStarted","Data":"21159e880c447d4370b0600ee203e9fad7c3b8e1bf35107aac4cb468533f5684"}
Sep 05 02:20:55.683520 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:55.683475    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55f695f754-tr5cj" podStartSLOduration=1.712390063 podStartE2EDuration="4.683462207s" totalImagesPullingTime="2.971072144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:20:55.683101067 +0000 UTC m=+159.238257376" watchObservedRunningTime="2026-09-05 02:20:55.683462207 +0000 UTC m=+159.238618513"
Sep 05 02:20:55.748208 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:55.748169    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:20:56.540790 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:56.540748    2557 patch_prober.go:28] interesting pod/image-registry-56897c4b7f-7m4jc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 05 02:20:56.541159 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:56.540807    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc" podUID="c3aa1a59-6ce1-4ae7-93d7-d680837d353d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:20:56.658513 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:20:56.658479    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:00.050872 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:00.050830    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:00.151421 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:00.151382    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:00.182714 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:00.182686    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:21:00.184884 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:00.184865    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19ece4dc-ca43-42a0-bf9c-64ab7e233025-metrics-tls\") pod \"dns-default-qs48v\" (UID: \"19ece4dc-ca43-42a0-bf9c-64ab7e233025\") " pod="openshift-dns/dns-default-qs48v"
Sep 05 02:21:00.469125 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:00.469097    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k9hcm\""
Sep 05 02:21:00.477199 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:00.477178    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qs48v"
Sep 05 02:21:00.783334 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:00.783260    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qs48v"]
Sep 05 02:21:00.786049 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:21:00.786024    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ece4dc_ca43_42a0_bf9c_64ab7e233025.slice/crio-84cba3ffcf43916922e4969419ee4ca25f155dffd634cbd524230f9c12ded959 WatchSource:0}: Error finding container 84cba3ffcf43916922e4969419ee4ca25f155dffd634cbd524230f9c12ded959: Status 404 returned error can't find the container with id 84cba3ffcf43916922e4969419ee4ca25f155dffd634cbd524230f9c12ded959
Sep 05 02:21:01.690871 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:01.690828    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qs48v" event={"ID":"19ece4dc-ca43-42a0-bf9c-64ab7e233025","Type":"ContainerStarted","Data":"84cba3ffcf43916922e4969419ee4ca25f155dffd634cbd524230f9c12ded959"}
Sep 05 02:21:01.984478 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:01.984378    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:21:01.984478 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:01.984469    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:21:01.986047 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:01.986018    2557 patch_prober.go:28] interesting pod/console-55f695f754-tr5cj container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused" start-of-body=
Sep 05 02:21:01.986192 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:01.986071    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-55f695f754-tr5cj" podUID="35134ac3-0010-441c-9adc-654aa2ac871a" containerName="console" probeResult="failure" output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused"
Sep 05 02:21:02.696140 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:02.696106    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qs48v" event={"ID":"19ece4dc-ca43-42a0-bf9c-64ab7e233025","Type":"ContainerStarted","Data":"03a7df60d7925a28d14a047c295cef8ca143c4f689a24a0f786701ef26796249"}
Sep 05 02:21:02.696537 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:02.696183    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qs48v" event={"ID":"19ece4dc-ca43-42a0-bf9c-64ab7e233025","Type":"ContainerStarted","Data":"39052c2621274aa47d4cd36d326ac518b5691622b816fea1f5959de4e8d5cc2f"}
Sep 05 02:21:02.696537 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:02.696255    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qs48v"
Sep 05 02:21:02.714247 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:02.714194    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qs48v" podStartSLOduration=129.001274739 podStartE2EDuration="2m10.714178271s" totalImagesPullingTime="1.712903532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:21:02.712923286 +0000 UTC m=+166.268079595" watchObservedRunningTime="2026-09-05 02:21:02.714178271 +0000 UTC m=+166.269334580"
Sep 05 02:21:04.840284 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:04.840249    2557 patch_prober.go:28] interesting pod/image-registry-56897c4b7f-7m4jc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 05 02:21:04.840660 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:04.840300    2557 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc" podUID="c3aa1a59-6ce1-4ae7-93d7-d680837d353d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:21:04.840660 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:04.840339    2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:21:04.840799 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:04.840769    2557 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"b30e32d2b3491f81e4c4cb2ccc999d6106e6f6600d367cb7d99c42dadea5c3a1"} pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc" containerMessage="Container registry failed liveness probe, will be restarted"
Sep 05 02:21:04.844299 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:04.844270    2557 patch_prober.go:28] interesting pod/image-registry-56897c4b7f-7m4jc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 05 02:21:04.844455 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:04.844311    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc" podUID="c3aa1a59-6ce1-4ae7-93d7-d680837d353d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:21:04.862332 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:04.862307    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:04.961793 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:04.961756    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:05.062144 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:05.062108    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:05.120983 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:05.120904    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:09.174858 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:09.174776    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:09.276255 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:09.276225    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:11.984168 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:11.984129    2557 patch_prober.go:28] interesting pod/console-55f695f754-tr5cj container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused" start-of-body=
Sep 05 02:21:11.984704 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:11.984199    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-55f695f754-tr5cj" podUID="35134ac3-0010-441c-9adc-654aa2ac871a" containerName="console" probeResult="failure" output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused"
Sep 05 02:21:12.701942 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:12.701910    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qs48v"
Sep 05 02:21:13.151798 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:13.151765    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:13.250843 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:13.250810    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:13.312826 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:13.312794    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:14.844566 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:14.844525    2557 patch_prober.go:28] interesting pod/image-registry-56897c4b7f-7m4jc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 05 02:21:14.844921 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:14.844575    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc" podUID="c3aa1a59-6ce1-4ae7-93d7-d680837d353d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:21:15.297612 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:15.297529    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:15.354745 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:15.354717    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:15.742926 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:15.742897    2557 generic.go:356] "Generic (PLEG): container finished" podID="d16b566d-44e4-46fe-bf78-6bb6df91a954" containerID="a6ac8f09092d5678ac60b98c84511d589520d1ceb3d71160845249683071430a" exitCode=0
Sep 05 02:21:15.743071 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:15.742932    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c" event={"ID":"d16b566d-44e4-46fe-bf78-6bb6df91a954","Type":"ContainerDied","Data":"a6ac8f09092d5678ac60b98c84511d589520d1ceb3d71160845249683071430a"}
Sep 05 02:21:15.743273 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:15.743256    2557 scope.go:117] "RemoveContainer" containerID="a6ac8f09092d5678ac60b98c84511d589520d1ceb3d71160845249683071430a"
Sep 05 02:21:16.748904 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:16.748869    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-tnh5c" event={"ID":"d16b566d-44e4-46fe-bf78-6bb6df91a954","Type":"ContainerStarted","Data":"f6d29f8b6e1ca6742fd3dd7c93bdafba0205fbcaf7abb437b3459941af0aea40"}
Sep 05 02:21:17.821480 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:17.821431    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:17.877135 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:17.877095    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:18.759755 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:18.759715    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:18.850763 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:18.850707    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:19.861264 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:19.861219    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:19.962697 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:19.962664    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:20.277299 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:20.277221    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55f695f754-tr5cj"]
Sep 05 02:21:20.401800 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:20.401760    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:20.494764 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:20.494739    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6dmz_d010c7b3-2a2d-4aef-b279-2948b7f0b0f9/init-textfile/0.log"
Sep 05 02:21:20.500645 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:20.500621    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:20.557247 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:20.557184    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:20.696938 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:20.696909    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6dmz_d010c7b3-2a2d-4aef-b279-2948b7f0b0f9/node-exporter/0.log"
Sep 05 02:21:20.768269 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:20.768235    2557 generic.go:356] "Generic (PLEG): container finished" podID="12667c29-3724-4078-964d-07268713477a" containerID="cb2558d992769f877b04edaccbcf6c3ff26aecc3c43c8d4fdb55a54692dbdb53" exitCode=0
Sep 05 02:21:20.768414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:20.768274    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4" event={"ID":"12667c29-3724-4078-964d-07268713477a","Type":"ContainerDied","Data":"cb2558d992769f877b04edaccbcf6c3ff26aecc3c43c8d4fdb55a54692dbdb53"}
Sep 05 02:21:20.768616 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:20.768602    2557 scope.go:117] "RemoveContainer" containerID="cb2558d992769f877b04edaccbcf6c3ff26aecc3c43c8d4fdb55a54692dbdb53"
Sep 05 02:21:20.895782 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:20.895750    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6dmz_d010c7b3-2a2d-4aef-b279-2948b7f0b0f9/kube-rbac-proxy/0.log"
Sep 05 02:21:20.985465 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:20.985415    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:21.082909 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:21.082875    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:21.773754 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:21.773723    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-54qc4" event={"ID":"12667c29-3724-4078-964d-07268713477a","Type":"ContainerStarted","Data":"d6e3cc636773790da5bd19d2cb8570384b7e257bfca4fcfa978b710bb6b36e53"}
Sep 05 02:21:21.896398 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:21.896363    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:22.002729 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:22.002700    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:22.101489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:22.101416    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:24.844698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:24.844665    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:21:24.983980 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:24.983943    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:25.086198 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:25.086167    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:25.095470 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:25.095375    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:21:25.184912 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:25.184881    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:25.303159 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:25.303125    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/3.log"
Sep 05 02:21:25.495393 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:25.495297    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-55f695f754-tr5cj_35134ac3-0010-441c-9adc-654aa2ac871a/console/0.log"
Sep 05 02:21:26.896103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:26.896070    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qs48v_19ece4dc-ca43-42a0-bf9c-64ab7e233025/dns/0.log"
Sep 05 02:21:26.904603 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:26.904576    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:26.971374 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:26.971348    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:27.095619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:27.095587    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qs48v_19ece4dc-ca43-42a0-bf9c-64ab7e233025/kube-rbac-proxy/0.log"
Sep 05 02:21:27.695170 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:27.695143    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g8xt5_c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b/dns-node-resolver/0.log"
Sep 05 02:21:29.860120 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:29.860081    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc" podUID="c3aa1a59-6ce1-4ae7-93d7-d680837d353d" containerName="registry" containerID="cri-o://b30e32d2b3491f81e4c4cb2ccc999d6106e6f6600d367cb7d99c42dadea5c3a1" gracePeriod=30
Sep 05 02:21:30.862314 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:30.862279    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:30.934497 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:30.934460    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:31.811951 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:31.811916    2557 generic.go:356] "Generic (PLEG): container finished" podID="c3aa1a59-6ce1-4ae7-93d7-d680837d353d" containerID="b30e32d2b3491f81e4c4cb2ccc999d6106e6f6600d367cb7d99c42dadea5c3a1" exitCode=0
Sep 05 02:21:31.812137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:31.811983    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc" event={"ID":"c3aa1a59-6ce1-4ae7-93d7-d680837d353d","Type":"ContainerDied","Data":"b30e32d2b3491f81e4c4cb2ccc999d6106e6f6600d367cb7d99c42dadea5c3a1"}
Sep 05 02:21:31.812137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:31.812017    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc" event={"ID":"c3aa1a59-6ce1-4ae7-93d7-d680837d353d","Type":"ContainerStarted","Data":"492c4279ee6b338d778cf878b44828e50f419c642d967ce5fc3ea0eb0895bf1a"}
Sep 05 02:21:31.812137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:31.812056    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:21:32.064819 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:32.064716    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:32.120827 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:32.120785    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:34.865578 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:34.865544    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:34.966709 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:34.966663    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:45.296604 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.296542    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55f695f754-tr5cj" podUID="35134ac3-0010-441c-9adc-654aa2ac871a" containerName="console" containerID="cri-o://21159e880c447d4370b0600ee203e9fad7c3b8e1bf35107aac4cb468533f5684" gracePeriod=15
Sep 05 02:21:45.538857 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.538832    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:21:45.625359 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.625334    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35134ac3-0010-441c-9adc-654aa2ac871a-console-oauth-config\") pod \"35134ac3-0010-441c-9adc-654aa2ac871a\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") "
Sep 05 02:21:45.625533 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.625365    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-console-config\") pod \"35134ac3-0010-441c-9adc-654aa2ac871a\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") "
Sep 05 02:21:45.625533 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.625395    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-oauth-serving-cert\") pod \"35134ac3-0010-441c-9adc-654aa2ac871a\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") "
Sep 05 02:21:45.625533 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.625423    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ws78\" (UniqueName: \"kubernetes.io/projected/35134ac3-0010-441c-9adc-654aa2ac871a-kube-api-access-9ws78\") pod \"35134ac3-0010-441c-9adc-654aa2ac871a\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") "
Sep 05 02:21:45.625533 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.625467    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35134ac3-0010-441c-9adc-654aa2ac871a-console-serving-cert\") pod \"35134ac3-0010-441c-9adc-654aa2ac871a\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") "
Sep 05 02:21:45.625533 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.625489    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-service-ca\") pod \"35134ac3-0010-441c-9adc-654aa2ac871a\" (UID: \"35134ac3-0010-441c-9adc-654aa2ac871a\") "
Sep 05 02:21:45.625846 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.625815    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-console-config" (OuterVolumeSpecName: "console-config") pod "35134ac3-0010-441c-9adc-654aa2ac871a" (UID: "35134ac3-0010-441c-9adc-654aa2ac871a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:45.625946 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.625833    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "35134ac3-0010-441c-9adc-654aa2ac871a" (UID: "35134ac3-0010-441c-9adc-654aa2ac871a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:45.625946 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.625894    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-service-ca" (OuterVolumeSpecName: "service-ca") pod "35134ac3-0010-441c-9adc-654aa2ac871a" (UID: "35134ac3-0010-441c-9adc-654aa2ac871a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:45.627716 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.627689    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35134ac3-0010-441c-9adc-654aa2ac871a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "35134ac3-0010-441c-9adc-654aa2ac871a" (UID: "35134ac3-0010-441c-9adc-654aa2ac871a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:45.627716 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.627706    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35134ac3-0010-441c-9adc-654aa2ac871a-kube-api-access-9ws78" (OuterVolumeSpecName: "kube-api-access-9ws78") pod "35134ac3-0010-441c-9adc-654aa2ac871a" (UID: "35134ac3-0010-441c-9adc-654aa2ac871a"). InnerVolumeSpecName "kube-api-access-9ws78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:21:45.627821 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.627715    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35134ac3-0010-441c-9adc-654aa2ac871a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "35134ac3-0010-441c-9adc-654aa2ac871a" (UID: "35134ac3-0010-441c-9adc-654aa2ac871a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:45.726805 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.726776    2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35134ac3-0010-441c-9adc-654aa2ac871a-console-oauth-config\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:21:45.726805 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.726801    2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-console-config\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:21:45.726805 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.726811    2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-oauth-serving-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:21:45.726988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.726820    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ws78\" (UniqueName: \"kubernetes.io/projected/35134ac3-0010-441c-9adc-654aa2ac871a-kube-api-access-9ws78\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:21:45.726988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.726829    2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35134ac3-0010-441c-9adc-654aa2ac871a-console-serving-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:21:45.726988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.726837    2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35134ac3-0010-441c-9adc-654aa2ac871a-service-ca\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:21:45.863254 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.863224    2557 generic.go:356] "Generic (PLEG): container finished" podID="35134ac3-0010-441c-9adc-654aa2ac871a" containerID="21159e880c447d4370b0600ee203e9fad7c3b8e1bf35107aac4cb468533f5684" exitCode=0
Sep 05 02:21:45.863388 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.863266    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f695f754-tr5cj"
Sep 05 02:21:45.863388 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.863273    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f695f754-tr5cj" event={"ID":"35134ac3-0010-441c-9adc-654aa2ac871a","Type":"ContainerDied","Data":"21159e880c447d4370b0600ee203e9fad7c3b8e1bf35107aac4cb468533f5684"}
Sep 05 02:21:45.863388 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.863340    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f695f754-tr5cj" event={"ID":"35134ac3-0010-441c-9adc-654aa2ac871a","Type":"ContainerDied","Data":"38972e0592f247c5c46ab9ef5ad3f613008b5fdc8262584710f414da20f25e93"}
Sep 05 02:21:45.863388 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.863360    2557 scope.go:117] "RemoveContainer" containerID="21159e880c447d4370b0600ee203e9fad7c3b8e1bf35107aac4cb468533f5684"
Sep 05 02:21:45.872103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.872084    2557 scope.go:117] "RemoveContainer" containerID="21159e880c447d4370b0600ee203e9fad7c3b8e1bf35107aac4cb468533f5684"
Sep 05 02:21:45.872381 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:21:45.872363    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21159e880c447d4370b0600ee203e9fad7c3b8e1bf35107aac4cb468533f5684\": container with ID starting with 21159e880c447d4370b0600ee203e9fad7c3b8e1bf35107aac4cb468533f5684 not found: ID does not exist" containerID="21159e880c447d4370b0600ee203e9fad7c3b8e1bf35107aac4cb468533f5684"
Sep 05 02:21:45.872418 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.872390    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21159e880c447d4370b0600ee203e9fad7c3b8e1bf35107aac4cb468533f5684"} err="failed to get container status \"21159e880c447d4370b0600ee203e9fad7c3b8e1bf35107aac4cb468533f5684\": rpc error: code = NotFound desc = could not find container \"21159e880c447d4370b0600ee203e9fad7c3b8e1bf35107aac4cb468533f5684\": container with ID starting with 21159e880c447d4370b0600ee203e9fad7c3b8e1bf35107aac4cb468533f5684 not found: ID does not exist"
Sep 05 02:21:45.884472 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.884413    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55f695f754-tr5cj"]
Sep 05 02:21:45.888052 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:45.888032    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55f695f754-tr5cj"]
Sep 05 02:21:46.988762 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:46.988730    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35134ac3-0010-441c-9adc-654aa2ac871a" path="/var/lib/kubelet/pods/35134ac3-0010-441c-9adc-654aa2ac871a/volumes"
Sep 05 02:21:49.862949 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:49.862918    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:49.916652 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:49.916615    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:52.820501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:52.820468    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56897c4b7f-7m4jc"
Sep 05 02:21:53.005806 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:53.005775    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:53.065527 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:53.065492    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:57.595716 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:57.595674    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:21:57.697391 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:21:57.697357    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:02.685297 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.685262    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-78667cd95d-g9snm"]
Sep 05 02:22:02.685766 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.685528    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35134ac3-0010-441c-9adc-654aa2ac871a" containerName="console"
Sep 05 02:22:02.685766 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.685540    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="35134ac3-0010-441c-9adc-654aa2ac871a" containerName="console"
Sep 05 02:22:02.685766 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.685599    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="35134ac3-0010-441c-9adc-654aa2ac871a" containerName="console"
Sep 05 02:22:02.689990 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.689969    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.692199 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.692175    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 05 02:22:02.692325 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.692286    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 05 02:22:02.692420 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.692396    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 05 02:22:02.692420 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.692412    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 05 02:22:02.692637 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.692419    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lldxf\""
Sep 05 02:22:02.692637 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.692420    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 05 02:22:02.697893 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.697876    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 05 02:22:02.703949 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.703926    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78667cd95d-g9snm"]
Sep 05 02:22:02.737274 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.737241    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11424145-7ce8-41b5-8bdb-44046dfdded2-federate-client-tls\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.737432 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.737279    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11424145-7ce8-41b5-8bdb-44046dfdded2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.737432 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.737301    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11424145-7ce8-41b5-8bdb-44046dfdded2-serving-certs-ca-bundle\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.737432 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.737378    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11424145-7ce8-41b5-8bdb-44046dfdded2-telemeter-client-tls\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.737432 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.737423    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11424145-7ce8-41b5-8bdb-44046dfdded2-secret-telemeter-client\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.737619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.737454    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11424145-7ce8-41b5-8bdb-44046dfdded2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.737619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.737495    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11424145-7ce8-41b5-8bdb-44046dfdded2-metrics-client-ca\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.737619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.737513    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nck5x\" (UniqueName: \"kubernetes.io/projected/11424145-7ce8-41b5-8bdb-44046dfdded2-kube-api-access-nck5x\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.838907 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.838873    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11424145-7ce8-41b5-8bdb-44046dfdded2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.838907 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.838911    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11424145-7ce8-41b5-8bdb-44046dfdded2-serving-certs-ca-bundle\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.839147 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.838937    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11424145-7ce8-41b5-8bdb-44046dfdded2-telemeter-client-tls\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.839147 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.838959    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11424145-7ce8-41b5-8bdb-44046dfdded2-secret-telemeter-client\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.839147 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.838974    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11424145-7ce8-41b5-8bdb-44046dfdded2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.839147 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.838990    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11424145-7ce8-41b5-8bdb-44046dfdded2-metrics-client-ca\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.839147 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.839015    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nck5x\" (UniqueName: \"kubernetes.io/projected/11424145-7ce8-41b5-8bdb-44046dfdded2-kube-api-access-nck5x\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.839147 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.839092    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11424145-7ce8-41b5-8bdb-44046dfdded2-federate-client-tls\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.839751 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.839726    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11424145-7ce8-41b5-8bdb-44046dfdded2-serving-certs-ca-bundle\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.839920 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.839877    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11424145-7ce8-41b5-8bdb-44046dfdded2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.840039 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.839968    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11424145-7ce8-41b5-8bdb-44046dfdded2-metrics-client-ca\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.841602 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.841574    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11424145-7ce8-41b5-8bdb-44046dfdded2-secret-telemeter-client\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.841947 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.841924    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11424145-7ce8-41b5-8bdb-44046dfdded2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.842057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.842043    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11424145-7ce8-41b5-8bdb-44046dfdded2-telemeter-client-tls\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.842096 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.842070    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11424145-7ce8-41b5-8bdb-44046dfdded2-federate-client-tls\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:02.846588 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:02.846563    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nck5x\" (UniqueName: \"kubernetes.io/projected/11424145-7ce8-41b5-8bdb-44046dfdded2-kube-api-access-nck5x\") pod \"telemeter-client-78667cd95d-g9snm\" (UID: \"11424145-7ce8-41b5-8bdb-44046dfdded2\") " pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:03.000774 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:03.000695    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm"
Sep 05 02:22:03.280571 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:03.280544    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78667cd95d-g9snm"]
Sep 05 02:22:03.282788 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:22:03.282756    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11424145_7ce8_41b5_8bdb_44046dfdded2.slice/crio-8b20e174c2bb2996114590c580a879a148b2d29e74fc40086840f479a15773ee WatchSource:0}: Error finding container 8b20e174c2bb2996114590c580a879a148b2d29e74fc40086840f479a15773ee: Status 404 returned error can't find the container with id 8b20e174c2bb2996114590c580a879a148b2d29e74fc40086840f479a15773ee
Sep 05 02:22:03.931517 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:03.931483    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm" event={"ID":"11424145-7ce8-41b5-8bdb-44046dfdded2","Type":"ContainerStarted","Data":"8b20e174c2bb2996114590c580a879a148b2d29e74fc40086840f479a15773ee"}
Sep 05 02:22:04.874241 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:04.874200    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:04.975233 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:04.975188    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:05.940851 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:05.940814    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm" event={"ID":"11424145-7ce8-41b5-8bdb-44046dfdded2","Type":"ContainerStarted","Data":"c43a5d3db5b5aaf6f798a24a4027eed0ae4c4d723400dfd2e154a80e4083277d"}
Sep 05 02:22:06.945982 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:06.945944    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm" event={"ID":"11424145-7ce8-41b5-8bdb-44046dfdded2","Type":"ContainerStarted","Data":"04d06a5adcbfa1d0de2c3a8b1c669d5cb7fe975136c6d2361da55a5dcb36442d"}
Sep 05 02:22:06.946354 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:06.946000    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm" event={"ID":"11424145-7ce8-41b5-8bdb-44046dfdded2","Type":"ContainerStarted","Data":"459befdc1f3edc09242963fa74fe68cc84987d77a20547f0e8abc5d207102a96"}
Sep 05 02:22:06.968187 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:06.968136    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-78667cd95d-g9snm" podStartSLOduration=1.983653358 podStartE2EDuration="4.968121325s" totalImagesPullingTime="2.984467967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:22:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:22:06.966148631 +0000 UTC m=+230.521304940" watchObservedRunningTime="2026-09-05 02:22:06.968121325 +0000 UTC m=+230.523277634"
Sep 05 02:22:07.109162 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:07.109128    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:07.171329 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:07.171295    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:07.256279 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:07.256204    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:07.316111 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:07.316075    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:07.862910 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:07.862871    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:07.948784 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:07.948751    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:16.302857 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:16.302816    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:18.032715 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:18.032675    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:18.094262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:18.094231    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:18.174884 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:18.174848    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:19.860580 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:19.860540    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:19.931880 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:19.931843    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:22:46.240616 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:22:46.240533    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:12.835482 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:12.835415    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:12.893411 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:12.893380    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:16.831504 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:16.831471    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:23:16.831952 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:16.831933    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:23:16.843325 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:16.843302    2557 kubelet.go:1648] "Image garbage collection succeeded"
Sep 05 02:23:24.801829 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:24.801793    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:24.900604 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:24.900573    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:28.421122 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:28.421076    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:34.887217 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:34.887181    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:34.999245 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:34.999211    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:37.801842 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:37.801800    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:37.855099 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:37.855067    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:37.947818 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:37.947784    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:38.048351 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:38.048325    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:38.583344 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:38.583311    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:38.671349 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:38.671314    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:48.340757 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:48.340712    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:48.439709 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:48.439671    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:23:48.502574 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:23:48.502533    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:06.723932 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:06.723893    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:06.805555 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:06.805520    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:13.210535 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:13.210428    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:13.312338 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:13.309131    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:13.366874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:13.366837    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:14.084496 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.084464    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"]
Sep 05 02:24:14.087621 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.087606    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"
Sep 05 02:24:14.090595 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.090571    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 05 02:24:14.090699 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.090639    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pfjxp\""
Sep 05 02:24:14.091428 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.091411    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 05 02:24:14.095762 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.095736    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"]
Sep 05 02:24:14.215331 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.215305    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f72c59-6e8e-4661-9808-34f8ea0ebe27-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk\" (UID: \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"
Sep 05 02:24:14.215678 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.215356    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37f72c59-6e8e-4661-9808-34f8ea0ebe27-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk\" (UID: \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"
Sep 05 02:24:14.215678 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.215382    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zv9c\" (UniqueName: \"kubernetes.io/projected/37f72c59-6e8e-4661-9808-34f8ea0ebe27-kube-api-access-7zv9c\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk\" (UID: \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"
Sep 05 02:24:14.316122 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.316094    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37f72c59-6e8e-4661-9808-34f8ea0ebe27-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk\" (UID: \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"
Sep 05 02:24:14.316283 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.316132    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zv9c\" (UniqueName: \"kubernetes.io/projected/37f72c59-6e8e-4661-9808-34f8ea0ebe27-kube-api-access-7zv9c\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk\" (UID: \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"
Sep 05 02:24:14.316283 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.316166    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f72c59-6e8e-4661-9808-34f8ea0ebe27-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk\" (UID: \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"
Sep 05 02:24:14.316486 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.316460    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37f72c59-6e8e-4661-9808-34f8ea0ebe27-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk\" (UID: \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"
Sep 05 02:24:14.316561 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.316497    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f72c59-6e8e-4661-9808-34f8ea0ebe27-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk\" (UID: \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"
Sep 05 02:24:14.324291 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.324265    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zv9c\" (UniqueName: \"kubernetes.io/projected/37f72c59-6e8e-4661-9808-34f8ea0ebe27-kube-api-access-7zv9c\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk\" (UID: \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"
Sep 05 02:24:14.396895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.396835    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"
Sep 05 02:24:14.665461 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.665408    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"]
Sep 05 02:24:14.667818 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:24:14.667787    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f72c59_6e8e_4661_9808_34f8ea0ebe27.slice/crio-485092a088a68e04d475042138c2057162610c1b134f319cccb74bcbace465ba WatchSource:0}: Error finding container 485092a088a68e04d475042138c2057162610c1b134f319cccb74bcbace465ba: Status 404 returned error can't find the container with id 485092a088a68e04d475042138c2057162610c1b134f319cccb74bcbace465ba
Sep 05 02:24:14.669634 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:14.669617    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:24:15.390147 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:15.390107    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk" event={"ID":"37f72c59-6e8e-4661-9808-34f8ea0ebe27","Type":"ContainerStarted","Data":"485092a088a68e04d475042138c2057162610c1b134f319cccb74bcbace465ba"}
Sep 05 02:24:20.414902 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:20.414866    2557 generic.go:356] "Generic (PLEG): container finished" podID="37f72c59-6e8e-4661-9808-34f8ea0ebe27" containerID="f3868ecc51bb7f0ff42eb76aa8c4c3a7086b6f303d5c09f5fef2ce221da98d81" exitCode=0
Sep 05 02:24:20.415284 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:20.414917    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk" event={"ID":"37f72c59-6e8e-4661-9808-34f8ea0ebe27","Type":"ContainerDied","Data":"f3868ecc51bb7f0ff42eb76aa8c4c3a7086b6f303d5c09f5fef2ce221da98d81"}
Sep 05 02:24:22.425075 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:22.424988    2557 generic.go:356] "Generic (PLEG): container finished" podID="37f72c59-6e8e-4661-9808-34f8ea0ebe27" containerID="a82d84b2912fd7e52993de7a8f1ac88d1143db8c7abafc9039e0b2b9c4ee72c6" exitCode=0
Sep 05 02:24:22.425417 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:22.425069    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk" event={"ID":"37f72c59-6e8e-4661-9808-34f8ea0ebe27","Type":"ContainerDied","Data":"a82d84b2912fd7e52993de7a8f1ac88d1143db8c7abafc9039e0b2b9c4ee72c6"}
Sep 05 02:24:29.455237 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:29.455198    2557 generic.go:356] "Generic (PLEG): container finished" podID="37f72c59-6e8e-4661-9808-34f8ea0ebe27" containerID="69d0b7af4e967b37e823ea7e241e15ee403ee60a694426181db781151e6e9099" exitCode=0
Sep 05 02:24:29.455630 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:29.455282    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk" event={"ID":"37f72c59-6e8e-4661-9808-34f8ea0ebe27","Type":"ContainerDied","Data":"69d0b7af4e967b37e823ea7e241e15ee403ee60a694426181db781151e6e9099"}
Sep 05 02:24:30.584639 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:30.584612    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"
Sep 05 02:24:30.639009 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:30.638971    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37f72c59-6e8e-4661-9808-34f8ea0ebe27-util\") pod \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\" (UID: \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\") "
Sep 05 02:24:30.639167 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:30.639032    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zv9c\" (UniqueName: \"kubernetes.io/projected/37f72c59-6e8e-4661-9808-34f8ea0ebe27-kube-api-access-7zv9c\") pod \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\" (UID: \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\") "
Sep 05 02:24:30.639167 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:30.639079    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f72c59-6e8e-4661-9808-34f8ea0ebe27-bundle\") pod \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\" (UID: \"37f72c59-6e8e-4661-9808-34f8ea0ebe27\") "
Sep 05 02:24:30.639811 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:30.639779    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f72c59-6e8e-4661-9808-34f8ea0ebe27-bundle" (OuterVolumeSpecName: "bundle") pod "37f72c59-6e8e-4661-9808-34f8ea0ebe27" (UID: "37f72c59-6e8e-4661-9808-34f8ea0ebe27"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:24:30.641279 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:30.641256    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f72c59-6e8e-4661-9808-34f8ea0ebe27-kube-api-access-7zv9c" (OuterVolumeSpecName: "kube-api-access-7zv9c") pod "37f72c59-6e8e-4661-9808-34f8ea0ebe27" (UID: "37f72c59-6e8e-4661-9808-34f8ea0ebe27"). InnerVolumeSpecName "kube-api-access-7zv9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:24:30.643341 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:30.643320    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f72c59-6e8e-4661-9808-34f8ea0ebe27-util" (OuterVolumeSpecName: "util") pod "37f72c59-6e8e-4661-9808-34f8ea0ebe27" (UID: "37f72c59-6e8e-4661-9808-34f8ea0ebe27"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:24:30.740380 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:30.740299    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37f72c59-6e8e-4661-9808-34f8ea0ebe27-util\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:24:30.740380 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:30.740328    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zv9c\" (UniqueName: \"kubernetes.io/projected/37f72c59-6e8e-4661-9808-34f8ea0ebe27-kube-api-access-7zv9c\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:24:30.740380 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:30.740343    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f72c59-6e8e-4661-9808-34f8ea0ebe27-bundle\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:24:31.465368 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:31.465330    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk" event={"ID":"37f72c59-6e8e-4661-9808-34f8ea0ebe27","Type":"ContainerDied","Data":"485092a088a68e04d475042138c2057162610c1b134f319cccb74bcbace465ba"}
Sep 05 02:24:31.465368 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:31.465363    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apvszk"
Sep 05 02:24:31.465640 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:31.465409    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="485092a088a68e04d475042138c2057162610c1b134f319cccb74bcbace465ba"
Sep 05 02:24:35.847570 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:35.847535    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:37.487294 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:37.487258    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:38.389574 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.389532    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn"]
Sep 05 02:24:38.389868 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.389854    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f72c59-6e8e-4661-9808-34f8ea0ebe27" containerName="pull"
Sep 05 02:24:38.389917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.389870    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f72c59-6e8e-4661-9808-34f8ea0ebe27" containerName="pull"
Sep 05 02:24:38.389917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.389878    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f72c59-6e8e-4661-9808-34f8ea0ebe27" containerName="extract"
Sep 05 02:24:38.389917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.389885    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f72c59-6e8e-4661-9808-34f8ea0ebe27" containerName="extract"
Sep 05 02:24:38.389917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.389896    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f72c59-6e8e-4661-9808-34f8ea0ebe27" containerName="util"
Sep 05 02:24:38.389917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.389902    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f72c59-6e8e-4661-9808-34f8ea0ebe27" containerName="util"
Sep 05 02:24:38.390066 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.389952    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f72c59-6e8e-4661-9808-34f8ea0ebe27" containerName="extract"
Sep 05 02:24:38.434559 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.434530    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn"
Sep 05 02:24:38.437166 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.437138    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn"]
Sep 05 02:24:38.446276 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.446249    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 05 02:24:38.446391 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.446317    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 05 02:24:38.446464 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.446409    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 05 02:24:38.447014 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.446996    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-vsnt8\""
Sep 05 02:24:38.492985 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.492954    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7xf4\" (UniqueName: \"kubernetes.io/projected/b83fca72-b68e-4543-8d6b-38e1ba9eb2e3-kube-api-access-k7xf4\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xmwnn\" (UID: \"b83fca72-b68e-4543-8d6b-38e1ba9eb2e3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn"
Sep 05 02:24:38.493321 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.493029    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b83fca72-b68e-4543-8d6b-38e1ba9eb2e3-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xmwnn\" (UID: \"b83fca72-b68e-4543-8d6b-38e1ba9eb2e3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn"
Sep 05 02:24:38.594290 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.594261    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b83fca72-b68e-4543-8d6b-38e1ba9eb2e3-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xmwnn\" (UID: \"b83fca72-b68e-4543-8d6b-38e1ba9eb2e3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn"
Sep 05 02:24:38.594480 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.594296    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7xf4\" (UniqueName: \"kubernetes.io/projected/b83fca72-b68e-4543-8d6b-38e1ba9eb2e3-kube-api-access-k7xf4\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xmwnn\" (UID: \"b83fca72-b68e-4543-8d6b-38e1ba9eb2e3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn"
Sep 05 02:24:38.596580 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.596554    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b83fca72-b68e-4543-8d6b-38e1ba9eb2e3-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xmwnn\" (UID: \"b83fca72-b68e-4543-8d6b-38e1ba9eb2e3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn"
Sep 05 02:24:38.604691 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.604667    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7xf4\" (UniqueName: \"kubernetes.io/projected/b83fca72-b68e-4543-8d6b-38e1ba9eb2e3-kube-api-access-k7xf4\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xmwnn\" (UID: \"b83fca72-b68e-4543-8d6b-38e1ba9eb2e3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn"
Sep 05 02:24:38.745148 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:38.745066    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn"
Sep 05 02:24:39.040700 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:39.040627    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn"]
Sep 05 02:24:39.043850 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:24:39.043820    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83fca72_b68e_4543_8d6b_38e1ba9eb2e3.slice/crio-63472c9215aff87b8d006b0f33f563829644debe58df5551a3fc21799bfa4b20 WatchSource:0}: Error finding container 63472c9215aff87b8d006b0f33f563829644debe58df5551a3fc21799bfa4b20: Status 404 returned error can't find the container with id 63472c9215aff87b8d006b0f33f563829644debe58df5551a3fc21799bfa4b20
Sep 05 02:24:39.494633 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:39.494600    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn" event={"ID":"b83fca72-b68e-4543-8d6b-38e1ba9eb2e3","Type":"ContainerStarted","Data":"63472c9215aff87b8d006b0f33f563829644debe58df5551a3fc21799bfa4b20"}
Sep 05 02:24:43.140809 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.140774    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:43.476333 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.476255    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-wmx7c"]
Sep 05 02:24:43.480109 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.480070    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:43.495140 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.495120    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-wmx7c"]
Sep 05 02:24:43.495260 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.495215    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:24:43.502453 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.498170    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 05 02:24:43.502453 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.498408    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 05 02:24:43.502453 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.498791    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-7td29\""
Sep 05 02:24:43.513382 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.513353    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn" event={"ID":"b83fca72-b68e-4543-8d6b-38e1ba9eb2e3","Type":"ContainerStarted","Data":"9314d19abd1e761124dc963b8247816f6470b63dd860b25c7816f0493e873443"}
Sep 05 02:24:43.513511 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.513462    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn"
Sep 05 02:24:43.534091 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.534065    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f406710a-1866-477b-aab8-df4f72ba4049-certificates\") pod \"keda-operator-579c8958f7-wmx7c\" (UID: \"f406710a-1866-477b-aab8-df4f72ba4049\") " pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:24:43.534197 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.534096    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f406710a-1866-477b-aab8-df4f72ba4049-cabundle0\") pod \"keda-operator-579c8958f7-wmx7c\" (UID: \"f406710a-1866-477b-aab8-df4f72ba4049\") " pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:24:43.534262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.534195    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lnc6\" (UniqueName: \"kubernetes.io/projected/f406710a-1866-477b-aab8-df4f72ba4049-kube-api-access-6lnc6\") pod \"keda-operator-579c8958f7-wmx7c\" (UID: \"f406710a-1866-477b-aab8-df4f72ba4049\") " pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:24:43.538811 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.538771    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn" podStartSLOduration=2.126109717 podStartE2EDuration="5.53875821s" totalImagesPullingTime="3.412648493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:24:43.536632185 +0000 UTC m=+387.091788493" watchObservedRunningTime="2026-09-05 02:24:43.53875821 +0000 UTC m=+387.093914519"
Sep 05 02:24:43.635385 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.635350    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lnc6\" (UniqueName: \"kubernetes.io/projected/f406710a-1866-477b-aab8-df4f72ba4049-kube-api-access-6lnc6\") pod \"keda-operator-579c8958f7-wmx7c\" (UID: \"f406710a-1866-477b-aab8-df4f72ba4049\") " pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:24:43.635541 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.635411    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f406710a-1866-477b-aab8-df4f72ba4049-certificates\") pod \"keda-operator-579c8958f7-wmx7c\" (UID: \"f406710a-1866-477b-aab8-df4f72ba4049\") " pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:24:43.635541 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.635431    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f406710a-1866-477b-aab8-df4f72ba4049-cabundle0\") pod \"keda-operator-579c8958f7-wmx7c\" (UID: \"f406710a-1866-477b-aab8-df4f72ba4049\") " pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:24:43.635622 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:24:43.635539    2557 secret.go:281] references non-existent secret key: ca.crt
Sep 05 02:24:43.635622 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:24:43.635558    2557 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 05 02:24:43.635622 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:24:43.635567    2557 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-wmx7c: references non-existent secret key: ca.crt
Sep 05 02:24:43.635729 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:24:43.635625    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f406710a-1866-477b-aab8-df4f72ba4049-certificates podName:f406710a-1866-477b-aab8-df4f72ba4049 nodeName:}" failed. No retries permitted until 2026-09-05 02:24:44.135610821 +0000 UTC m=+387.690767108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f406710a-1866-477b-aab8-df4f72ba4049-certificates") pod "keda-operator-579c8958f7-wmx7c" (UID: "f406710a-1866-477b-aab8-df4f72ba4049") : references non-existent secret key: ca.crt
Sep 05 02:24:43.636024 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.636007    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f406710a-1866-477b-aab8-df4f72ba4049-cabundle0\") pod \"keda-operator-579c8958f7-wmx7c\" (UID: \"f406710a-1866-477b-aab8-df4f72ba4049\") " pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:24:43.650788 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.650759    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lnc6\" (UniqueName: \"kubernetes.io/projected/f406710a-1866-477b-aab8-df4f72ba4049-kube-api-access-6lnc6\") pod \"keda-operator-579c8958f7-wmx7c\" (UID: \"f406710a-1866-477b-aab8-df4f72ba4049\") " pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:24:43.687071 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:43.687041    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:44.139202 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.139161    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f406710a-1866-477b-aab8-df4f72ba4049-certificates\") pod \"keda-operator-579c8958f7-wmx7c\" (UID: \"f406710a-1866-477b-aab8-df4f72ba4049\") " pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:24:44.139371 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:24:44.139324    2557 secret.go:281] references non-existent secret key: ca.crt
Sep 05 02:24:44.139371 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:24:44.139351    2557 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 05 02:24:44.139371 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:24:44.139363    2557 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-wmx7c: references non-existent secret key: ca.crt
Sep 05 02:24:44.139514 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:24:44.139455    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f406710a-1866-477b-aab8-df4f72ba4049-certificates podName:f406710a-1866-477b-aab8-df4f72ba4049 nodeName:}" failed. No retries permitted until 2026-09-05 02:24:45.139422156 +0000 UTC m=+388.694578458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f406710a-1866-477b-aab8-df4f72ba4049-certificates") pod "keda-operator-579c8958f7-wmx7c" (UID: "f406710a-1866-477b-aab8-df4f72ba4049") : references non-existent secret key: ca.crt
Sep 05 02:24:44.168539 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.168508    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:44.337426 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.337388    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"]
Sep 05 02:24:44.339646 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.339629    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"
Sep 05 02:24:44.342578 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.342550    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 05 02:24:44.357405 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.357376    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"]
Sep 05 02:24:44.432891 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.432802    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:44.440783 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.440756    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnr7n\" (UniqueName: \"kubernetes.io/projected/a2355c78-7452-4831-b465-e3138c1cd5de-kube-api-access-tnr7n\") pod \"keda-metrics-apiserver-77857b7b98-wpgn4\" (UID: \"a2355c78-7452-4831-b465-e3138c1cd5de\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"
Sep 05 02:24:44.440866 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.440804    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a2355c78-7452-4831-b465-e3138c1cd5de-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-wpgn4\" (UID: \"a2355c78-7452-4831-b465-e3138c1cd5de\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"
Sep 05 02:24:44.440929 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.440902    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a2355c78-7452-4831-b465-e3138c1cd5de-certificates\") pod \"keda-metrics-apiserver-77857b7b98-wpgn4\" (UID: \"a2355c78-7452-4831-b465-e3138c1cd5de\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"
Sep 05 02:24:44.541666 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.541627    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnr7n\" (UniqueName: \"kubernetes.io/projected/a2355c78-7452-4831-b465-e3138c1cd5de-kube-api-access-tnr7n\") pod \"keda-metrics-apiserver-77857b7b98-wpgn4\" (UID: \"a2355c78-7452-4831-b465-e3138c1cd5de\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"
Sep 05 02:24:44.541856 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.541703    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a2355c78-7452-4831-b465-e3138c1cd5de-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-wpgn4\" (UID: \"a2355c78-7452-4831-b465-e3138c1cd5de\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"
Sep 05 02:24:44.541856 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.541777    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a2355c78-7452-4831-b465-e3138c1cd5de-certificates\") pod \"keda-metrics-apiserver-77857b7b98-wpgn4\" (UID: \"a2355c78-7452-4831-b465-e3138c1cd5de\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"
Sep 05 02:24:44.542378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.542348    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a2355c78-7452-4831-b465-e3138c1cd5de-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-wpgn4\" (UID: \"a2355c78-7452-4831-b465-e3138c1cd5de\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"
Sep 05 02:24:44.544336 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.544312    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a2355c78-7452-4831-b465-e3138c1cd5de-certificates\") pod \"keda-metrics-apiserver-77857b7b98-wpgn4\" (UID: \"a2355c78-7452-4831-b465-e3138c1cd5de\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"
Sep 05 02:24:44.561226 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.561197    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnr7n\" (UniqueName: \"kubernetes.io/projected/a2355c78-7452-4831-b465-e3138c1cd5de-kube-api-access-tnr7n\") pod \"keda-metrics-apiserver-77857b7b98-wpgn4\" (UID: \"a2355c78-7452-4831-b465-e3138c1cd5de\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"
Sep 05 02:24:44.588847 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.588812    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:44.649425 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.649395    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"
Sep 05 02:24:44.821123 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.821088    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:44.938970 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:44.938938    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"]
Sep 05 02:24:44.942220 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:24:44.942194    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2355c78_7452_4831_b465_e3138c1cd5de.slice/crio-7999fd18f77f71be9a9681c6971f7a9962ee9be2939992c6882bcbe55880195b WatchSource:0}: Error finding container 7999fd18f77f71be9a9681c6971f7a9962ee9be2939992c6882bcbe55880195b: Status 404 returned error can't find the container with id 7999fd18f77f71be9a9681c6971f7a9962ee9be2939992c6882bcbe55880195b
Sep 05 02:24:45.039911 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:45.039880    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:45.146704 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:45.146673    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f406710a-1866-477b-aab8-df4f72ba4049-certificates\") pod \"keda-operator-579c8958f7-wmx7c\" (UID: \"f406710a-1866-477b-aab8-df4f72ba4049\") " pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:24:45.149074 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:45.149042    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f406710a-1866-477b-aab8-df4f72ba4049-certificates\") pod \"keda-operator-579c8958f7-wmx7c\" (UID: \"f406710a-1866-477b-aab8-df4f72ba4049\") " pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:24:45.261003 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:45.260920    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:45.312027 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:45.312000    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:24:45.373991 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:45.373962    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:45.522834 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:45.522760    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4" event={"ID":"a2355c78-7452-4831-b465-e3138c1cd5de","Type":"ContainerStarted","Data":"7999fd18f77f71be9a9681c6971f7a9962ee9be2939992c6882bcbe55880195b"}
Sep 05 02:24:45.605298 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:45.605265    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-wmx7c"]
Sep 05 02:24:45.611995 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:24:45.611967    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf406710a_1866_477b_aab8_df4f72ba4049.slice/crio-a0d0bfc77539bc99fb295786fd75af27ffac3f0a271a8368c82f8247a498920e WatchSource:0}: Error finding container a0d0bfc77539bc99fb295786fd75af27ffac3f0a271a8368c82f8247a498920e: Status 404 returned error can't find the container with id a0d0bfc77539bc99fb295786fd75af27ffac3f0a271a8368c82f8247a498920e
Sep 05 02:24:45.616127 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:45.616087    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:45.734210 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:45.734178    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:24:46.529235 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:46.529194    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-wmx7c" event={"ID":"f406710a-1866-477b-aab8-df4f72ba4049","Type":"ContainerStarted","Data":"a0d0bfc77539bc99fb295786fd75af27ffac3f0a271a8368c82f8247a498920e"}
Sep 05 02:24:49.544863 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:49.544817    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-wmx7c" event={"ID":"f406710a-1866-477b-aab8-df4f72ba4049","Type":"ContainerStarted","Data":"806b9a4bdf3ed7ab84756b4e7a682975fc13fe241f32f15ead9763b382bd3ac7"}
Sep 05 02:24:49.545292 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:49.544969    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:24:49.546749 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:49.546718    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4" event={"ID":"a2355c78-7452-4831-b465-e3138c1cd5de","Type":"ContainerStarted","Data":"ec5f7e945d97826e94321475f8e70d56f435a9f02356ea95c0bc1d65810234e6"}
Sep 05 02:24:49.546880 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:49.546847    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"
Sep 05 02:24:49.567346 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:49.567294    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-wmx7c" podStartSLOduration=3.421839151 podStartE2EDuration="6.567281065s" totalImagesPullingTime="3.145441914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:24:49.565852754 +0000 UTC m=+393.121009057" watchObservedRunningTime="2026-09-05 02:24:49.567281065 +0000 UTC m=+393.122437374"
Sep 05 02:24:49.586391 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:24:49.586340    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4" podStartSLOduration=1.7740635359999999 podStartE2EDuration="5.586324834s" totalImagesPullingTime="3.812261298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:24:49.584203843 +0000 UTC m=+393.139360152" watchObservedRunningTime="2026-09-05 02:24:49.586324834 +0000 UTC m=+393.141481143"
Sep 05 02:25:00.557925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:00.557894    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-wpgn4"
Sep 05 02:25:04.519950 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:04.519917    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xmwnn"
Sep 05 02:25:10.555948 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:10.555918    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-wmx7c"
Sep 05 02:25:34.658176 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:34.658139    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:34.810205 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:34.810165    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:35.212786 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.212748    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"]
Sep 05 02:25:35.219720 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.219696    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"
Sep 05 02:25:35.232059 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.232041    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 05 02:25:35.232416 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.232401    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 05 02:25:35.232477 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.232408    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pfjxp\""
Sep 05 02:25:35.251574 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.251548    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"]
Sep 05 02:25:35.358225 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.358177    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/781b78d3-c9da-467f-b976-ef21cca793d8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq\" (UID: \"781b78d3-c9da-467f-b976-ef21cca793d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"
Sep 05 02:25:35.358393 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.358284    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntf56\" (UniqueName: \"kubernetes.io/projected/781b78d3-c9da-467f-b976-ef21cca793d8-kube-api-access-ntf56\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq\" (UID: \"781b78d3-c9da-467f-b976-ef21cca793d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"
Sep 05 02:25:35.358393 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.358346    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/781b78d3-c9da-467f-b976-ef21cca793d8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq\" (UID: \"781b78d3-c9da-467f-b976-ef21cca793d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"
Sep 05 02:25:35.459177 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.459139    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/781b78d3-c9da-467f-b976-ef21cca793d8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq\" (UID: \"781b78d3-c9da-467f-b976-ef21cca793d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"
Sep 05 02:25:35.459334 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.459208    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntf56\" (UniqueName: \"kubernetes.io/projected/781b78d3-c9da-467f-b976-ef21cca793d8-kube-api-access-ntf56\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq\" (UID: \"781b78d3-c9da-467f-b976-ef21cca793d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"
Sep 05 02:25:35.459334 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.459249    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/781b78d3-c9da-467f-b976-ef21cca793d8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq\" (UID: \"781b78d3-c9da-467f-b976-ef21cca793d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"
Sep 05 02:25:35.459636 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.459613    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/781b78d3-c9da-467f-b976-ef21cca793d8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq\" (UID: \"781b78d3-c9da-467f-b976-ef21cca793d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"
Sep 05 02:25:35.459692 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.459649    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/781b78d3-c9da-467f-b976-ef21cca793d8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq\" (UID: \"781b78d3-c9da-467f-b976-ef21cca793d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"
Sep 05 02:25:35.471304 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.471252    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntf56\" (UniqueName: \"kubernetes.io/projected/781b78d3-c9da-467f-b976-ef21cca793d8-kube-api-access-ntf56\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq\" (UID: \"781b78d3-c9da-467f-b976-ef21cca793d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"
Sep 05 02:25:35.529100 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.529068    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"
Sep 05 02:25:35.820165 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:35.820009    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"]
Sep 05 02:25:35.822914 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:25:35.822884    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781b78d3_c9da_467f_b976_ef21cca793d8.slice/crio-33a2ff418c6a38418eb7910e929944cf03f035bc5233d9dcd96c3afe76bd5ab8 WatchSource:0}: Error finding container 33a2ff418c6a38418eb7910e929944cf03f035bc5233d9dcd96c3afe76bd5ab8: Status 404 returned error can't find the container with id 33a2ff418c6a38418eb7910e929944cf03f035bc5233d9dcd96c3afe76bd5ab8
Sep 05 02:25:36.750405 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:36.750326    2557 generic.go:356] "Generic (PLEG): container finished" podID="781b78d3-c9da-467f-b976-ef21cca793d8" containerID="dcf52a85664ef22dba32a4e7401b4913a48f603ab6a0ca36eb175b658d353aab" exitCode=0
Sep 05 02:25:36.750559 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:36.750416    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq" event={"ID":"781b78d3-c9da-467f-b976-ef21cca793d8","Type":"ContainerDied","Data":"dcf52a85664ef22dba32a4e7401b4913a48f603ab6a0ca36eb175b658d353aab"}
Sep 05 02:25:36.750559 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:36.750487    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq" event={"ID":"781b78d3-c9da-467f-b976-ef21cca793d8","Type":"ContainerStarted","Data":"33a2ff418c6a38418eb7910e929944cf03f035bc5233d9dcd96c3afe76bd5ab8"}
Sep 05 02:25:39.767672 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:39.767632    2557 generic.go:356] "Generic (PLEG): container finished" podID="781b78d3-c9da-467f-b976-ef21cca793d8" containerID="3f083dd76a0c64ad59dae88ce46bb9c83a786375e5360c02ac24e1e6738f010c" exitCode=0
Sep 05 02:25:39.768098 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:39.767713    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq" event={"ID":"781b78d3-c9da-467f-b976-ef21cca793d8","Type":"ContainerDied","Data":"3f083dd76a0c64ad59dae88ce46bb9c83a786375e5360c02ac24e1e6738f010c"}
Sep 05 02:25:40.774607 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:40.774570    2557 generic.go:356] "Generic (PLEG): container finished" podID="781b78d3-c9da-467f-b976-ef21cca793d8" containerID="6b0dd770fd91f94799f2fc5fdd475a6b86a2352f06d7df3a15397b6afd3412ce" exitCode=0
Sep 05 02:25:40.775022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:40.774625    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq" event={"ID":"781b78d3-c9da-467f-b976-ef21cca793d8","Type":"ContainerDied","Data":"6b0dd770fd91f94799f2fc5fdd475a6b86a2352f06d7df3a15397b6afd3412ce"}
Sep 05 02:25:41.904323 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:41.904300    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"
Sep 05 02:25:42.104540 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:42.104513    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/781b78d3-c9da-467f-b976-ef21cca793d8-bundle\") pod \"781b78d3-c9da-467f-b976-ef21cca793d8\" (UID: \"781b78d3-c9da-467f-b976-ef21cca793d8\") "
Sep 05 02:25:42.104706 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:42.104580    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntf56\" (UniqueName: \"kubernetes.io/projected/781b78d3-c9da-467f-b976-ef21cca793d8-kube-api-access-ntf56\") pod \"781b78d3-c9da-467f-b976-ef21cca793d8\" (UID: \"781b78d3-c9da-467f-b976-ef21cca793d8\") "
Sep 05 02:25:42.104706 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:42.104612    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/781b78d3-c9da-467f-b976-ef21cca793d8-util\") pod \"781b78d3-c9da-467f-b976-ef21cca793d8\" (UID: \"781b78d3-c9da-467f-b976-ef21cca793d8\") "
Sep 05 02:25:42.105252 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:42.105225    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781b78d3-c9da-467f-b976-ef21cca793d8-bundle" (OuterVolumeSpecName: "bundle") pod "781b78d3-c9da-467f-b976-ef21cca793d8" (UID: "781b78d3-c9da-467f-b976-ef21cca793d8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:25:42.106662 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:42.106634    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781b78d3-c9da-467f-b976-ef21cca793d8-kube-api-access-ntf56" (OuterVolumeSpecName: "kube-api-access-ntf56") pod "781b78d3-c9da-467f-b976-ef21cca793d8" (UID: "781b78d3-c9da-467f-b976-ef21cca793d8"). InnerVolumeSpecName "kube-api-access-ntf56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:25:42.109371 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:42.109335    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781b78d3-c9da-467f-b976-ef21cca793d8-util" (OuterVolumeSpecName: "util") pod "781b78d3-c9da-467f-b976-ef21cca793d8" (UID: "781b78d3-c9da-467f-b976-ef21cca793d8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:25:42.205381 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:42.205345    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/781b78d3-c9da-467f-b976-ef21cca793d8-util\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:25:42.205381 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:42.205373    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/781b78d3-c9da-467f-b976-ef21cca793d8-bundle\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:25:42.205381 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:42.205384    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntf56\" (UniqueName: \"kubernetes.io/projected/781b78d3-c9da-467f-b976-ef21cca793d8-kube-api-access-ntf56\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:25:42.784050 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:42.784020    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq"
Sep 05 02:25:42.784050 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:42.784035    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bmwhq" event={"ID":"781b78d3-c9da-467f-b976-ef21cca793d8","Type":"ContainerDied","Data":"33a2ff418c6a38418eb7910e929944cf03f035bc5233d9dcd96c3afe76bd5ab8"}
Sep 05 02:25:42.784247 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:42.784077    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33a2ff418c6a38418eb7910e929944cf03f035bc5233d9dcd96c3afe76bd5ab8"
Sep 05 02:25:46.620610 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:46.620578    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:46.949364 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:46.949272    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:47.758350 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:47.758311    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:48.152997 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:48.152958    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:49.548279 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:49.548239    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:49.698986 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:49.698946    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:52.677265 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:52.677231    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:52.932748 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:52.932661    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:53.430193 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.430160    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-868fdc5cdd-jtvkn"]
Sep 05 02:25:53.430461 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.430448    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781b78d3-c9da-467f-b976-ef21cca793d8" containerName="pull"
Sep 05 02:25:53.430508 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.430462    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="781b78d3-c9da-467f-b976-ef21cca793d8" containerName="pull"
Sep 05 02:25:53.430508 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.430477    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781b78d3-c9da-467f-b976-ef21cca793d8" containerName="extract"
Sep 05 02:25:53.430508 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.430483    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="781b78d3-c9da-467f-b976-ef21cca793d8" containerName="extract"
Sep 05 02:25:53.430508 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.430492    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781b78d3-c9da-467f-b976-ef21cca793d8" containerName="util"
Sep 05 02:25:53.430508 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.430497    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="781b78d3-c9da-467f-b976-ef21cca793d8" containerName="util"
Sep 05 02:25:53.430659 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.430543    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="781b78d3-c9da-467f-b976-ef21cca793d8" containerName="extract"
Sep 05 02:25:53.432673 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.432650    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.437488 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.437466    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 05 02:25:53.437614 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.437520    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 05 02:25:53.438052 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.438029    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 05 02:25:53.438155 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.438092    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 05 02:25:53.438201 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.438163    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 05 02:25:53.439274 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.439127    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 05 02:25:53.439274 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.439129    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 05 02:25:53.439404 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.439344    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m5q29\""
Sep 05 02:25:53.463005 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.462981    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 05 02:25:53.470861 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.470839    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868fdc5cdd-jtvkn"]
Sep 05 02:25:53.481597 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.481574    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-console-oauth-config\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.481684 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.481603    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-oauth-serving-cert\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.481726 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.481689    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zqc8\" (UniqueName: \"kubernetes.io/projected/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-kube-api-access-5zqc8\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.481764 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.481722    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-service-ca\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.481795 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.481763    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-trusted-ca-bundle\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.481795 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.481785    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-console-config\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.481867 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.481823    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-console-serving-cert\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.582371 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.582335    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-console-serving-cert\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.582371 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.582378    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-console-oauth-config\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.582620 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.582396    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-oauth-serving-cert\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.582620 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.582431    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zqc8\" (UniqueName: \"kubernetes.io/projected/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-kube-api-access-5zqc8\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.582620 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.582476    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-service-ca\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.582620 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.582511    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-trusted-ca-bundle\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.582620 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.582539    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-console-config\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.583194 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.583170    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-oauth-serving-cert\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.583324 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.583301    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-console-config\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.583391 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.583347    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-service-ca\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.583534 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.583515    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-trusted-ca-bundle\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.584966 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.584942    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-console-serving-cert\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.585051 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.584975    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-console-oauth-config\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.597010 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.596981    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zqc8\" (UniqueName: \"kubernetes.io/projected/db98bf51-1b00-40cc-ba86-3f7aef2f8a5b-kube-api-access-5zqc8\") pod \"console-868fdc5cdd-jtvkn\" (UID: \"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b\") " pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.602547 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.602519    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:53.741594 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.741496    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:25:53.763972 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:53.763942    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:54.043270 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:54.043197    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868fdc5cdd-jtvkn"]
Sep 05 02:25:54.046161 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:25:54.046126    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb98bf51_1b00_40cc_ba86_3f7aef2f8a5b.slice/crio-99ce1d5a2b3f544f019e793b76616ba6ea6f63e59bfc07e7c4042ae82b6457d7 WatchSource:0}: Error finding container 99ce1d5a2b3f544f019e793b76616ba6ea6f63e59bfc07e7c4042ae82b6457d7: Status 404 returned error can't find the container with id 99ce1d5a2b3f544f019e793b76616ba6ea6f63e59bfc07e7c4042ae82b6457d7
Sep 05 02:25:54.836846 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:54.836806    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868fdc5cdd-jtvkn" event={"ID":"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b","Type":"ContainerStarted","Data":"7bfb0b59a9472d616f500cc2f21a332c580ffd580a4dd5e0603162eceb22f896"}
Sep 05 02:25:54.837307 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:54.836861    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868fdc5cdd-jtvkn" event={"ID":"db98bf51-1b00-40cc-ba86-3f7aef2f8a5b","Type":"ContainerStarted","Data":"99ce1d5a2b3f544f019e793b76616ba6ea6f63e59bfc07e7c4042ae82b6457d7"}
Sep 05 02:25:54.886081 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:54.886028    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-868fdc5cdd-jtvkn" podStartSLOduration=1.8860128459999999 podStartE2EDuration="1.886012846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:25:54.884171255 +0000 UTC m=+458.439327563" watchObservedRunningTime="2026-09-05 02:25:54.886012846 +0000 UTC m=+458.441169155"
Sep 05 02:25:54.963512 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:54.963476    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:55.133563 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:55.133529    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:56.220430 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.220382    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:56.338239 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.338204    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:56.658947 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.658909    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"]
Sep 05 02:25:56.662038 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.662014    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"
Sep 05 02:25:56.667367 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.667338    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 05 02:25:56.667563 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.667554    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pfjxp\""
Sep 05 02:25:56.667632 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.667577    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 05 02:25:56.684613 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.684587    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"]
Sep 05 02:25:56.707264 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.707235    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89f845be-d652-499b-96ca-96fb244df88d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88\" (UID: \"89f845be-d652-499b-96ca-96fb244df88d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"
Sep 05 02:25:56.707386 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.707277    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89f845be-d652-499b-96ca-96fb244df88d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88\" (UID: \"89f845be-d652-499b-96ca-96fb244df88d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"
Sep 05 02:25:56.707386 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.707363    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl592\" (UniqueName: \"kubernetes.io/projected/89f845be-d652-499b-96ca-96fb244df88d-kube-api-access-dl592\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88\" (UID: \"89f845be-d652-499b-96ca-96fb244df88d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"
Sep 05 02:25:56.808331 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.808292    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl592\" (UniqueName: \"kubernetes.io/projected/89f845be-d652-499b-96ca-96fb244df88d-kube-api-access-dl592\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88\" (UID: \"89f845be-d652-499b-96ca-96fb244df88d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"
Sep 05 02:25:56.808522 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.808351    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89f845be-d652-499b-96ca-96fb244df88d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88\" (UID: \"89f845be-d652-499b-96ca-96fb244df88d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"
Sep 05 02:25:56.808522 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.808388    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89f845be-d652-499b-96ca-96fb244df88d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88\" (UID: \"89f845be-d652-499b-96ca-96fb244df88d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"
Sep 05 02:25:56.808850 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.808828    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89f845be-d652-499b-96ca-96fb244df88d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88\" (UID: \"89f845be-d652-499b-96ca-96fb244df88d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"
Sep 05 02:25:56.808886 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.808851    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89f845be-d652-499b-96ca-96fb244df88d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88\" (UID: \"89f845be-d652-499b-96ca-96fb244df88d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"
Sep 05 02:25:56.839807 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.839775    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl592\" (UniqueName: \"kubernetes.io/projected/89f845be-d652-499b-96ca-96fb244df88d-kube-api-access-dl592\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88\" (UID: \"89f845be-d652-499b-96ca-96fb244df88d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"
Sep 05 02:25:56.971419 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:56.971337    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"
Sep 05 02:25:57.251919 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:57.251892    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"]
Sep 05 02:25:57.254462 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:25:57.254421    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f845be_d652_499b_96ca_96fb244df88d.slice/crio-faae500015a079ce774d3a6290548515186e8490f5c8a1fa6425ca6e41f2668a WatchSource:0}: Error finding container faae500015a079ce774d3a6290548515186e8490f5c8a1fa6425ca6e41f2668a: Status 404 returned error can't find the container with id faae500015a079ce774d3a6290548515186e8490f5c8a1fa6425ca6e41f2668a
Sep 05 02:25:57.427992 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:57.427950    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:57.533956 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:57.533872    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:57.851772 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:57.851739    2557 generic.go:356] "Generic (PLEG): container finished" podID="89f845be-d652-499b-96ca-96fb244df88d" containerID="dce884e4084ebb2516dd7aa6d9097e21f610512afa0a1448476aa418e6dc010b" exitCode=0
Sep 05 02:25:57.851880 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:57.851781    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88" event={"ID":"89f845be-d652-499b-96ca-96fb244df88d","Type":"ContainerDied","Data":"dce884e4084ebb2516dd7aa6d9097e21f610512afa0a1448476aa418e6dc010b"}
Sep 05 02:25:57.851880 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:57.851813    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88" event={"ID":"89f845be-d652-499b-96ca-96fb244df88d","Type":"ContainerStarted","Data":"faae500015a079ce774d3a6290548515186e8490f5c8a1fa6425ca6e41f2668a"}
Sep 05 02:25:58.409521 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:58.409486    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:25:58.502119 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:25:58.502086    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:00.874003 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:00.873971    2557 generic.go:356] "Generic (PLEG): container finished" podID="89f845be-d652-499b-96ca-96fb244df88d" containerID="69454c5b6d6b5435733de8dbfd90b8218e078c6481330f15997f316e10ffe330" exitCode=0
Sep 05 02:26:00.874387 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:00.874019    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88" event={"ID":"89f845be-d652-499b-96ca-96fb244df88d","Type":"ContainerDied","Data":"69454c5b6d6b5435733de8dbfd90b8218e078c6481330f15997f316e10ffe330"}
Sep 05 02:26:01.880111 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:01.880071    2557 generic.go:356] "Generic (PLEG): container finished" podID="89f845be-d652-499b-96ca-96fb244df88d" containerID="fc6bf52d80af2487a55917125336d6015e250b910712a23302b24a4973a6d69b" exitCode=0
Sep 05 02:26:01.880515 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:01.880118    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88" event={"ID":"89f845be-d652-499b-96ca-96fb244df88d","Type":"ContainerDied","Data":"fc6bf52d80af2487a55917125336d6015e250b910712a23302b24a4973a6d69b"}
Sep 05 02:26:02.050573 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:02.050519    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:02.226740 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:02.226634    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:03.006398 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.006373    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"
Sep 05 02:26:03.059644 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.059613    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89f845be-d652-499b-96ca-96fb244df88d-util\") pod \"89f845be-d652-499b-96ca-96fb244df88d\" (UID: \"89f845be-d652-499b-96ca-96fb244df88d\") "
Sep 05 02:26:03.059814 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.059669    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89f845be-d652-499b-96ca-96fb244df88d-bundle\") pod \"89f845be-d652-499b-96ca-96fb244df88d\" (UID: \"89f845be-d652-499b-96ca-96fb244df88d\") "
Sep 05 02:26:03.059814 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.059728    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl592\" (UniqueName: \"kubernetes.io/projected/89f845be-d652-499b-96ca-96fb244df88d-kube-api-access-dl592\") pod \"89f845be-d652-499b-96ca-96fb244df88d\" (UID: \"89f845be-d652-499b-96ca-96fb244df88d\") "
Sep 05 02:26:03.060163 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.060127    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f845be-d652-499b-96ca-96fb244df88d-bundle" (OuterVolumeSpecName: "bundle") pod "89f845be-d652-499b-96ca-96fb244df88d" (UID: "89f845be-d652-499b-96ca-96fb244df88d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:26:03.061878 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.061842    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f845be-d652-499b-96ca-96fb244df88d-kube-api-access-dl592" (OuterVolumeSpecName: "kube-api-access-dl592") pod "89f845be-d652-499b-96ca-96fb244df88d" (UID: "89f845be-d652-499b-96ca-96fb244df88d"). InnerVolumeSpecName "kube-api-access-dl592". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:26:03.063595 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.063574    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f845be-d652-499b-96ca-96fb244df88d-util" (OuterVolumeSpecName: "util") pod "89f845be-d652-499b-96ca-96fb244df88d" (UID: "89f845be-d652-499b-96ca-96fb244df88d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:26:03.160608 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.160497    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89f845be-d652-499b-96ca-96fb244df88d-bundle\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:03.160608 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.160549    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl592\" (UniqueName: \"kubernetes.io/projected/89f845be-d652-499b-96ca-96fb244df88d-kube-api-access-dl592\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:03.160608 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.160563    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89f845be-d652-499b-96ca-96fb244df88d-util\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:03.742062 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.742001    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:26:03.742273 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.742134    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:26:03.747140 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.747112    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:26:03.890739 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.890708    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88"
Sep 05 02:26:03.890903 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.890733    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47r88" event={"ID":"89f845be-d652-499b-96ca-96fb244df88d","Type":"ContainerDied","Data":"faae500015a079ce774d3a6290548515186e8490f5c8a1fa6425ca6e41f2668a"}
Sep 05 02:26:03.890903 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.890781    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faae500015a079ce774d3a6290548515186e8490f5c8a1fa6425ca6e41f2668a"
Sep 05 02:26:03.894683 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:03.894662    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-868fdc5cdd-jtvkn"
Sep 05 02:26:04.106982 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:04.106948    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:04.207796 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:04.207762    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:04.309849 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:04.309813    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:05.085472 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:05.085420    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:07.478782 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:07.478748    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:07.579478 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:07.579431    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:19.720309 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:19.720274    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:24.411688 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.411649    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:24.416291 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.416263    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"]
Sep 05 02:26:24.416567 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.416554    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f845be-d652-499b-96ca-96fb244df88d" containerName="pull"
Sep 05 02:26:24.416619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.416569    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f845be-d652-499b-96ca-96fb244df88d" containerName="pull"
Sep 05 02:26:24.416619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.416578    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f845be-d652-499b-96ca-96fb244df88d" containerName="extract"
Sep 05 02:26:24.416619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.416584    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f845be-d652-499b-96ca-96fb244df88d" containerName="extract"
Sep 05 02:26:24.416619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.416603    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f845be-d652-499b-96ca-96fb244df88d" containerName="util"
Sep 05 02:26:24.416619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.416608    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f845be-d652-499b-96ca-96fb244df88d" containerName="util"
Sep 05 02:26:24.416765 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.416659    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f845be-d652-499b-96ca-96fb244df88d" containerName="extract"
Sep 05 02:26:24.425313 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.425295    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.428075 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.428056    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 05 02:26:24.428933 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.428912    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 05 02:26:24.429028 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.428951    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 05 02:26:24.429028 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.428951    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 05 02:26:24.429135 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.428956    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 05 02:26:24.429215 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.429199    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 05 02:26:24.429513 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.429498    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-j9gq7\""
Sep 05 02:26:24.436866 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.436844    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"]
Sep 05 02:26:24.509284 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.509245    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:24.535010 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.534964    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.535010 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.535019    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.535197 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.535050    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wbms\" (UniqueName: \"kubernetes.io/projected/30f09e1f-2bf1-4837-891f-cab2ea128a2a-kube-api-access-5wbms\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.535197 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.535121    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.535197 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.535156    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.535197 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.535179    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.535347 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.535248    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f09e1f-2bf1-4837-891f-cab2ea128a2a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.636625 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.636588    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.636793 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.636630    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.636793 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.636670    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f09e1f-2bf1-4837-891f-cab2ea128a2a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.636793 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.636709    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.636793 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.636734    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.636793 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.636755    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wbms\" (UniqueName: \"kubernetes.io/projected/30f09e1f-2bf1-4837-891f-cab2ea128a2a-kube-api-access-5wbms\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.637104 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.636816    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.637471 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.637424    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.639115 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.639087    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f09e1f-2bf1-4837-891f-cab2ea128a2a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.639115 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.639103    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.639246 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.639123    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.639246 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.639227    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.644662 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.644641    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.644904 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.644881    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wbms\" (UniqueName: \"kubernetes.io/projected/30f09e1f-2bf1-4837-891f-cab2ea128a2a-kube-api-access-5wbms\") pod \"istiod-openshift-gateway-7b7995d5cd-kj7ds\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:24.734260 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:24.734177    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:25.012032 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:25.012006    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"]
Sep 05 02:26:25.014412 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:26:25.014384    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f09e1f_2bf1_4837_891f_cab2ea128a2a.slice/crio-b02aa0deec1469eb65548b6522c09a4875f5df2bce840682b375d10ffcea213b WatchSource:0}: Error finding container b02aa0deec1469eb65548b6522c09a4875f5df2bce840682b375d10ffcea213b: Status 404 returned error can't find the container with id b02aa0deec1469eb65548b6522c09a4875f5df2bce840682b375d10ffcea213b
Sep 05 02:26:25.454473 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:25.454410    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:25.986768 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:25.986725    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds" event={"ID":"30f09e1f-2bf1-4837-891f-cab2ea128a2a","Type":"ContainerStarted","Data":"b02aa0deec1469eb65548b6522c09a4875f5df2bce840682b375d10ffcea213b"}
Sep 05 02:26:27.380720 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:27.380676    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 02:26:27.381038 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:27.380748    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 02:26:27.997896 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:27.997856    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds" event={"ID":"30f09e1f-2bf1-4837-891f-cab2ea128a2a","Type":"ContainerStarted","Data":"49822b4f17de37a5e0babc81d5313b764c18ca922018816bc7dca729df49846d"}
Sep 05 02:26:27.998072 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:27.997982    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:28.025831 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:28.025781    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds" podStartSLOduration=1.661756408 podStartE2EDuration="4.025766499s" totalImagesPullingTime="2.364010091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:28.023797914 +0000 UTC m=+491.578954223" watchObservedRunningTime="2026-09-05 02:26:28.025766499 +0000 UTC m=+491.580922799"
Sep 05 02:26:29.003994 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:29.003955    2557 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-kj7ds container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 05 02:26:29.004380 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:29.004016    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds" podUID="30f09e1f-2bf1-4837-891f-cab2ea128a2a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:26:29.203092 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:29.203048    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:29.302699 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:29.302615    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:31.820686 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:31.820655    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:31.960067 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:31.960030    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:32.002899 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.002872    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:26:32.499341 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.499305    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"]
Sep 05 02:26:32.502697 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.502670    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.507115 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.507091    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-w7b27\""
Sep 05 02:26:32.518807 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.518781    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"]
Sep 05 02:26:32.556157 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.556125    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:32.603031 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.603003    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82df5585-ce83-449d-8931-c22fd607d1f4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.603165 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.603060    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.603218 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.603166    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.603274 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.603245    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnvjs\" (UniqueName: \"kubernetes.io/projected/82df5585-ce83-449d-8931-c22fd607d1f4-kube-api-access-cnvjs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.603317 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.603285    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.603361 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.603340    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82df5585-ce83-449d-8931-c22fd607d1f4-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.603397 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.603369    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.603460 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.603397    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82df5585-ce83-449d-8931-c22fd607d1f4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.603460 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.603426    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.646301 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.646276    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:32.704362 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.704327    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnvjs\" (UniqueName: \"kubernetes.io/projected/82df5585-ce83-449d-8931-c22fd607d1f4-kube-api-access-cnvjs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.704521 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.704371    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.704521 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.704394    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82df5585-ce83-449d-8931-c22fd607d1f4-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.704521 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.704414    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.704521 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.704463    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82df5585-ce83-449d-8931-c22fd607d1f4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.704521 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.704493    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.704779 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.704526    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82df5585-ce83-449d-8931-c22fd607d1f4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.704779 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.704574    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.704779 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.704628    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.704940 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.704915    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.705014 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.704993    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.705074 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.705036    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.705129 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.705094    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.705341 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.705314    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82df5585-ce83-449d-8931-c22fd607d1f4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.706762 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.706744    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82df5585-ce83-449d-8931-c22fd607d1f4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.706906 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.706891    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82df5585-ce83-449d-8931-c22fd607d1f4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.722085 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.722058    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnvjs\" (UniqueName: \"kubernetes.io/projected/82df5585-ce83-449d-8931-c22fd607d1f4-kube-api-access-cnvjs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.725626 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.725603    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82df5585-ce83-449d-8931-c22fd607d1f4-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm\" (UID: \"82df5585-ce83-449d-8931-c22fd607d1f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:32.813184 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:32.813101    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:33.091736 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:33.091705    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"]
Sep 05 02:26:33.093788 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:26:33.093759    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82df5585_ce83_449d_8931_c22fd607d1f4.slice/crio-2b94cdfe8735d268c5f94e419a2fcf54d195ddf925141155592fd318b40add65 WatchSource:0}: Error finding container 2b94cdfe8735d268c5f94e419a2fcf54d195ddf925141155592fd318b40add65: Status 404 returned error can't find the container with id 2b94cdfe8735d268c5f94e419a2fcf54d195ddf925141155592fd318b40add65
Sep 05 02:26:34.025729 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:34.025691    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm" event={"ID":"82df5585-ce83-449d-8931-c22fd607d1f4","Type":"ContainerStarted","Data":"2b94cdfe8735d268c5f94e419a2fcf54d195ddf925141155592fd318b40add65"}
Sep 05 02:26:35.765491 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:35.765413    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 02:26:35.765752 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:35.765534    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 02:26:35.765752 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:35.765563    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 02:26:36.037294 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:36.037192    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm" event={"ID":"82df5585-ce83-449d-8931-c22fd607d1f4","Type":"ContainerStarted","Data":"2f56ec957f9e0ef6f5f2467f41f36b343ed5d62cba84f9f40b5d918d4d4f4599"}
Sep 05 02:26:36.062207 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:36.062143    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm" podStartSLOduration=1.3926850100000001 podStartE2EDuration="4.062125128s" totalImagesPullingTime="2.669440118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:36.060701344 +0000 UTC m=+499.615857676" watchObservedRunningTime="2026-09-05 02:26:36.062125128 +0000 UTC m=+499.617281438"
Sep 05 02:26:36.314581 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:36.314550    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:36.814220 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:36.814181    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:36.819040 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:36.819013    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:37.041621 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:37.041589    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:37.042611 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:37.042593    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:26:38.027315 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:38.027268    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:38.127718 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:38.127680    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:44.827187 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:44.827153    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"]
Sep 05 02:26:44.830622 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:44.830604    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"
Sep 05 02:26:44.834521 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:44.834496    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 05 02:26:44.834671 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:44.834566    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pfjxp\""
Sep 05 02:26:44.834748 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:44.834732    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 05 02:26:44.842059 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:44.842036    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"]
Sep 05 02:26:44.914016 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:44.913979    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5gk8\" (UniqueName: \"kubernetes.io/projected/adf46c1f-2692-47b9-a657-ea92346ccc64-kube-api-access-d5gk8\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7\" (UID: \"adf46c1f-2692-47b9-a657-ea92346ccc64\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"
Sep 05 02:26:44.914216 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:44.914022    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adf46c1f-2692-47b9-a657-ea92346ccc64-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7\" (UID: \"adf46c1f-2692-47b9-a657-ea92346ccc64\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"
Sep 05 02:26:44.914216 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:44.914116    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adf46c1f-2692-47b9-a657-ea92346ccc64-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7\" (UID: \"adf46c1f-2692-47b9-a657-ea92346ccc64\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"
Sep 05 02:26:44.923788 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:44.923758    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"]
Sep 05 02:26:44.926942 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:44.926925    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"
Sep 05 02:26:44.936863 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:44.936836    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"]
Sep 05 02:26:45.014842 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.014807    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm\" (UID: \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"
Sep 05 02:26:45.015017 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.014860    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5gk8\" (UniqueName: \"kubernetes.io/projected/adf46c1f-2692-47b9-a657-ea92346ccc64-kube-api-access-d5gk8\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7\" (UID: \"adf46c1f-2692-47b9-a657-ea92346ccc64\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"
Sep 05 02:26:45.015017 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.014909    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm\" (UID: \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"
Sep 05 02:26:45.015017 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.014946    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tsps\" (UniqueName: \"kubernetes.io/projected/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-kube-api-access-8tsps\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm\" (UID: \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"
Sep 05 02:26:45.015017 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.014980    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adf46c1f-2692-47b9-a657-ea92346ccc64-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7\" (UID: \"adf46c1f-2692-47b9-a657-ea92346ccc64\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"
Sep 05 02:26:45.015214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.015080    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adf46c1f-2692-47b9-a657-ea92346ccc64-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7\" (UID: \"adf46c1f-2692-47b9-a657-ea92346ccc64\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"
Sep 05 02:26:45.015331 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.015315    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adf46c1f-2692-47b9-a657-ea92346ccc64-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7\" (UID: \"adf46c1f-2692-47b9-a657-ea92346ccc64\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"
Sep 05 02:26:45.015414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.015397    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adf46c1f-2692-47b9-a657-ea92346ccc64-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7\" (UID: \"adf46c1f-2692-47b9-a657-ea92346ccc64\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"
Sep 05 02:26:45.023786 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.023758    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"]
Sep 05 02:26:45.027452 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.027409    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5gk8\" (UniqueName: \"kubernetes.io/projected/adf46c1f-2692-47b9-a657-ea92346ccc64-kube-api-access-d5gk8\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7\" (UID: \"adf46c1f-2692-47b9-a657-ea92346ccc64\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"
Sep 05 02:26:45.029329 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.029305    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"
Sep 05 02:26:45.035875 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.035848    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"]
Sep 05 02:26:45.115940 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.115851    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5412088f-ac7e-4d5f-9223-783c5142b4e3-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc\" (UID: \"5412088f-ac7e-4d5f-9223-783c5142b4e3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"
Sep 05 02:26:45.115940 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.115906    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm\" (UID: \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"
Sep 05 02:26:45.116154 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.115992    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm\" (UID: \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"
Sep 05 02:26:45.116154 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.116036    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tsps\" (UniqueName: \"kubernetes.io/projected/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-kube-api-access-8tsps\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm\" (UID: \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"
Sep 05 02:26:45.116154 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.116068    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5412088f-ac7e-4d5f-9223-783c5142b4e3-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc\" (UID: \"5412088f-ac7e-4d5f-9223-783c5142b4e3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"
Sep 05 02:26:45.116154 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.116095    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stlh6\" (UniqueName: \"kubernetes.io/projected/5412088f-ac7e-4d5f-9223-783c5142b4e3-kube-api-access-stlh6\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc\" (UID: \"5412088f-ac7e-4d5f-9223-783c5142b4e3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"
Sep 05 02:26:45.116306 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.116258    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm\" (UID: \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"
Sep 05 02:26:45.116306 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.116292    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm\" (UID: \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"
Sep 05 02:26:45.121194 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.121170    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"]
Sep 05 02:26:45.124583 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.124568    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"
Sep 05 02:26:45.129998 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.129977    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tsps\" (UniqueName: \"kubernetes.io/projected/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-kube-api-access-8tsps\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm\" (UID: \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"
Sep 05 02:26:45.135056 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.135032    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"]
Sep 05 02:26:45.140248 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.140228    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"
Sep 05 02:26:45.217026 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.216991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5412088f-ac7e-4d5f-9223-783c5142b4e3-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc\" (UID: \"5412088f-ac7e-4d5f-9223-783c5142b4e3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"
Sep 05 02:26:45.217190 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.217034    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stlh6\" (UniqueName: \"kubernetes.io/projected/5412088f-ac7e-4d5f-9223-783c5142b4e3-kube-api-access-stlh6\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc\" (UID: \"5412088f-ac7e-4d5f-9223-783c5142b4e3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"
Sep 05 02:26:45.217190 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.217071    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxlf2\" (UniqueName: \"kubernetes.io/projected/aa67bfeb-b431-4745-add1-5d03ed865c59-kube-api-access-jxlf2\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j\" (UID: \"aa67bfeb-b431-4745-add1-5d03ed865c59\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"
Sep 05 02:26:45.217190 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.217110    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa67bfeb-b431-4745-add1-5d03ed865c59-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j\" (UID: \"aa67bfeb-b431-4745-add1-5d03ed865c59\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"
Sep 05 02:26:45.217190 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.217187    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5412088f-ac7e-4d5f-9223-783c5142b4e3-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc\" (UID: \"5412088f-ac7e-4d5f-9223-783c5142b4e3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"
Sep 05 02:26:45.217416 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.217209    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa67bfeb-b431-4745-add1-5d03ed865c59-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j\" (UID: \"aa67bfeb-b431-4745-add1-5d03ed865c59\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"
Sep 05 02:26:45.217577 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.217417    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5412088f-ac7e-4d5f-9223-783c5142b4e3-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc\" (UID: \"5412088f-ac7e-4d5f-9223-783c5142b4e3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"
Sep 05 02:26:45.217816 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.217797    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5412088f-ac7e-4d5f-9223-783c5142b4e3-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc\" (UID: \"5412088f-ac7e-4d5f-9223-783c5142b4e3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"
Sep 05 02:26:45.231055 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.231031    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stlh6\" (UniqueName: \"kubernetes.io/projected/5412088f-ac7e-4d5f-9223-783c5142b4e3-kube-api-access-stlh6\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc\" (UID: \"5412088f-ac7e-4d5f-9223-783c5142b4e3\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"
Sep 05 02:26:45.235755 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.235723    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"
Sep 05 02:26:45.318336 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.318301    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxlf2\" (UniqueName: \"kubernetes.io/projected/aa67bfeb-b431-4745-add1-5d03ed865c59-kube-api-access-jxlf2\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j\" (UID: \"aa67bfeb-b431-4745-add1-5d03ed865c59\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"
Sep 05 02:26:45.318502 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.318355    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa67bfeb-b431-4745-add1-5d03ed865c59-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j\" (UID: \"aa67bfeb-b431-4745-add1-5d03ed865c59\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"
Sep 05 02:26:45.318502 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.318448    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa67bfeb-b431-4745-add1-5d03ed865c59-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j\" (UID: \"aa67bfeb-b431-4745-add1-5d03ed865c59\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"
Sep 05 02:26:45.318784 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.318763    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa67bfeb-b431-4745-add1-5d03ed865c59-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j\" (UID: \"aa67bfeb-b431-4745-add1-5d03ed865c59\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"
Sep 05 02:26:45.318853 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.318832    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa67bfeb-b431-4745-add1-5d03ed865c59-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j\" (UID: \"aa67bfeb-b431-4745-add1-5d03ed865c59\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"
Sep 05 02:26:45.328864 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.328836    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxlf2\" (UniqueName: \"kubernetes.io/projected/aa67bfeb-b431-4745-add1-5d03ed865c59-kube-api-access-jxlf2\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j\" (UID: \"aa67bfeb-b431-4745-add1-5d03ed865c59\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"
Sep 05 02:26:45.343426 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.343407    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"
Sep 05 02:26:45.416333 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.416302    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"]
Sep 05 02:26:45.419734 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:26:45.419710    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf46c1f_2692_47b9_a657_ea92346ccc64.slice/crio-f109a03239b603abc7036992653172c8a5a2b6ef0b8b84c5ba5c1287c15ad886 WatchSource:0}: Error finding container f109a03239b603abc7036992653172c8a5a2b6ef0b8b84c5ba5c1287c15ad886: Status 404 returned error can't find the container with id f109a03239b603abc7036992653172c8a5a2b6ef0b8b84c5ba5c1287c15ad886
Sep 05 02:26:45.440529 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.440504    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"
Sep 05 02:26:45.508737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.508702    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"]
Sep 05 02:26:45.510944 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:26:45.510897    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b7fe46_ff99_45d6_a4a6_a7ec52c3d79d.slice/crio-a6aa72852fff40177722d54a2c5ddb125b30b921623dde1a527c378b4316d36f WatchSource:0}: Error finding container a6aa72852fff40177722d54a2c5ddb125b30b921623dde1a527c378b4316d36f: Status 404 returned error can't find the container with id a6aa72852fff40177722d54a2c5ddb125b30b921623dde1a527c378b4316d36f
Sep 05 02:26:45.615185 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.615162    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"]
Sep 05 02:26:45.616937 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:26:45.616909    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5412088f_ac7e_4d5f_9223_783c5142b4e3.slice/crio-8d7988d95cf6e0501be339fdfc34ea55e2d17dbcf0548f574a60c7b23d3b31ae WatchSource:0}: Error finding container 8d7988d95cf6e0501be339fdfc34ea55e2d17dbcf0548f574a60c7b23d3b31ae: Status 404 returned error can't find the container with id 8d7988d95cf6e0501be339fdfc34ea55e2d17dbcf0548f574a60c7b23d3b31ae
Sep 05 02:26:45.715710 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:45.715554    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"]
Sep 05 02:26:45.721904 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:26:45.721880    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa67bfeb_b431_4745_add1_5d03ed865c59.slice/crio-61ef585379eafde87e9bcfe5921ad190bc6e915f0177f1fa9de43c4b9c1bf46f WatchSource:0}: Error finding container 61ef585379eafde87e9bcfe5921ad190bc6e915f0177f1fa9de43c4b9c1bf46f: Status 404 returned error can't find the container with id 61ef585379eafde87e9bcfe5921ad190bc6e915f0177f1fa9de43c4b9c1bf46f
Sep 05 02:26:46.081840 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:46.081805    2557 generic.go:356] "Generic (PLEG): container finished" podID="53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d" containerID="b07ea62bf39a4f2b1341cd47e9f016fcadd873ce387b03679bb5138a952d622d" exitCode=0
Sep 05 02:26:46.082266 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:46.081889    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm" event={"ID":"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d","Type":"ContainerDied","Data":"b07ea62bf39a4f2b1341cd47e9f016fcadd873ce387b03679bb5138a952d622d"}
Sep 05 02:26:46.082266 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:46.081944    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm" event={"ID":"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d","Type":"ContainerStarted","Data":"a6aa72852fff40177722d54a2c5ddb125b30b921623dde1a527c378b4316d36f"}
Sep 05 02:26:46.083591 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:46.083536    2557 generic.go:356] "Generic (PLEG): container finished" podID="aa67bfeb-b431-4745-add1-5d03ed865c59" containerID="9c30fc36cb092fa6cd28c312ce0563fb29d85525715e7ff49995391c68b0c8b1" exitCode=0
Sep 05 02:26:46.083654 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:46.083597    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j" event={"ID":"aa67bfeb-b431-4745-add1-5d03ed865c59","Type":"ContainerDied","Data":"9c30fc36cb092fa6cd28c312ce0563fb29d85525715e7ff49995391c68b0c8b1"}
Sep 05 02:26:46.083654 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:46.083628    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j" event={"ID":"aa67bfeb-b431-4745-add1-5d03ed865c59","Type":"ContainerStarted","Data":"61ef585379eafde87e9bcfe5921ad190bc6e915f0177f1fa9de43c4b9c1bf46f"}
Sep 05 02:26:46.085428 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:46.085382    2557 generic.go:356] "Generic (PLEG): container finished" podID="5412088f-ac7e-4d5f-9223-783c5142b4e3" containerID="40640937d36979736ac10e14aa9c925f6c21cc6d390e32b0923ed39e7e156c80" exitCode=0
Sep 05 02:26:46.085553 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:46.085454    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc" event={"ID":"5412088f-ac7e-4d5f-9223-783c5142b4e3","Type":"ContainerDied","Data":"40640937d36979736ac10e14aa9c925f6c21cc6d390e32b0923ed39e7e156c80"}
Sep 05 02:26:46.085553 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:46.085506    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc" event={"ID":"5412088f-ac7e-4d5f-9223-783c5142b4e3","Type":"ContainerStarted","Data":"8d7988d95cf6e0501be339fdfc34ea55e2d17dbcf0548f574a60c7b23d3b31ae"}
Sep 05 02:26:46.087131 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:46.087111    2557 generic.go:356] "Generic (PLEG): container finished" podID="adf46c1f-2692-47b9-a657-ea92346ccc64" containerID="ba5bbb8ea23701c828f4c770f9ef33d03a7edea9a1da8e0f15079820902d06b7" exitCode=0
Sep 05 02:26:46.087210 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:46.087134    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7" event={"ID":"adf46c1f-2692-47b9-a657-ea92346ccc64","Type":"ContainerDied","Data":"ba5bbb8ea23701c828f4c770f9ef33d03a7edea9a1da8e0f15079820902d06b7"}
Sep 05 02:26:46.087262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:46.087167    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7" event={"ID":"adf46c1f-2692-47b9-a657-ea92346ccc64","Type":"ContainerStarted","Data":"f109a03239b603abc7036992653172c8a5a2b6ef0b8b84c5ba5c1287c15ad886"}
Sep 05 02:26:48.099535 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:48.099497    2557 generic.go:356] "Generic (PLEG): container finished" podID="53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d" containerID="b690cde6813efeba2ebeee00b95321b9ff5fb58beb8dc22aabca4fe3fabae923" exitCode=0
Sep 05 02:26:48.100008 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:48.099588    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm" event={"ID":"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d","Type":"ContainerDied","Data":"b690cde6813efeba2ebeee00b95321b9ff5fb58beb8dc22aabca4fe3fabae923"}
Sep 05 02:26:48.101525 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:48.101501    2557 generic.go:356] "Generic (PLEG): container finished" podID="aa67bfeb-b431-4745-add1-5d03ed865c59" containerID="35a6507c9113ad840e0783ca7ba9486dd03848d26ccde5c52cba8014e766140c" exitCode=0
Sep 05 02:26:48.101607 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:48.101583    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j" event={"ID":"aa67bfeb-b431-4745-add1-5d03ed865c59","Type":"ContainerDied","Data":"35a6507c9113ad840e0783ca7ba9486dd03848d26ccde5c52cba8014e766140c"}
Sep 05 02:26:48.103469 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:48.103430    2557 generic.go:356] "Generic (PLEG): container finished" podID="5412088f-ac7e-4d5f-9223-783c5142b4e3" containerID="a7618db630647313138f628ee10472c7c7cdd8c7a192cb3aee06f5812ed6fcb5" exitCode=0
Sep 05 02:26:48.103565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:48.103514    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc" event={"ID":"5412088f-ac7e-4d5f-9223-783c5142b4e3","Type":"ContainerDied","Data":"a7618db630647313138f628ee10472c7c7cdd8c7a192cb3aee06f5812ed6fcb5"}
Sep 05 02:26:48.108242 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:48.108216    2557 generic.go:356] "Generic (PLEG): container finished" podID="adf46c1f-2692-47b9-a657-ea92346ccc64" containerID="0e2224c938f5fce598926e5bd8d995a253b21cba7dda8fbe9e281e157d53ae38" exitCode=0
Sep 05 02:26:48.108338 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:48.108260    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7" event={"ID":"adf46c1f-2692-47b9-a657-ea92346ccc64","Type":"ContainerDied","Data":"0e2224c938f5fce598926e5bd8d995a253b21cba7dda8fbe9e281e157d53ae38"}
Sep 05 02:26:49.115281 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:49.115240    2557 generic.go:356] "Generic (PLEG): container finished" podID="53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d" containerID="d0c67f59ddb9a9aa7d9fa59f44793bc6ff1292b3b750a02f00600f750c9613d2" exitCode=0
Sep 05 02:26:49.115772 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:49.115315    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm" event={"ID":"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d","Type":"ContainerDied","Data":"d0c67f59ddb9a9aa7d9fa59f44793bc6ff1292b3b750a02f00600f750c9613d2"}
Sep 05 02:26:49.117412 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:49.117391    2557 generic.go:356] "Generic (PLEG): container finished" podID="aa67bfeb-b431-4745-add1-5d03ed865c59" containerID="e0db86afb6baa332bd480c4db730d2296438638d9b59c5884c59879cbcf905f3" exitCode=0
Sep 05 02:26:49.117521 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:49.117467    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j" event={"ID":"aa67bfeb-b431-4745-add1-5d03ed865c59","Type":"ContainerDied","Data":"e0db86afb6baa332bd480c4db730d2296438638d9b59c5884c59879cbcf905f3"}
Sep 05 02:26:49.119693 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:49.119670    2557 generic.go:356] "Generic (PLEG): container finished" podID="5412088f-ac7e-4d5f-9223-783c5142b4e3" containerID="b12e6b0e0f4bf8e293fa830b88f4fefa6ae4e114843b976699c71163ff99de8f" exitCode=0
Sep 05 02:26:49.119798 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:49.119741    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc" event={"ID":"5412088f-ac7e-4d5f-9223-783c5142b4e3","Type":"ContainerDied","Data":"b12e6b0e0f4bf8e293fa830b88f4fefa6ae4e114843b976699c71163ff99de8f"}
Sep 05 02:26:49.121768 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:49.121744    2557 generic.go:356] "Generic (PLEG): container finished" podID="adf46c1f-2692-47b9-a657-ea92346ccc64" containerID="0d4e9f040950be2e94c1a922b8eb3c8deb9e212104803c41189443c9730a1abf" exitCode=0
Sep 05 02:26:49.121876 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:49.121821    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7" event={"ID":"adf46c1f-2692-47b9-a657-ea92346ccc64","Type":"ContainerDied","Data":"0d4e9f040950be2e94c1a922b8eb3c8deb9e212104803c41189443c9730a1abf"}
Sep 05 02:26:49.849065 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:49.849029    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:50.254841 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.254815    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"
Sep 05 02:26:50.310250 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.310219    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"
Sep 05 02:26:50.325149 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.325126    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"
Sep 05 02:26:50.333225 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.333200    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"
Sep 05 02:26:50.362353 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.362322    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-bundle\") pod \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\" (UID: \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\") "
Sep 05 02:26:50.362531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.362362    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-util\") pod \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\" (UID: \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\") "
Sep 05 02:26:50.362531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.362382    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa67bfeb-b431-4745-add1-5d03ed865c59-util\") pod \"aa67bfeb-b431-4745-add1-5d03ed865c59\" (UID: \"aa67bfeb-b431-4745-add1-5d03ed865c59\") "
Sep 05 02:26:50.362531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.362397    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5412088f-ac7e-4d5f-9223-783c5142b4e3-bundle\") pod \"5412088f-ac7e-4d5f-9223-783c5142b4e3\" (UID: \"5412088f-ac7e-4d5f-9223-783c5142b4e3\") "
Sep 05 02:26:50.362531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.362417    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stlh6\" (UniqueName: \"kubernetes.io/projected/5412088f-ac7e-4d5f-9223-783c5142b4e3-kube-api-access-stlh6\") pod \"5412088f-ac7e-4d5f-9223-783c5142b4e3\" (UID: \"5412088f-ac7e-4d5f-9223-783c5142b4e3\") "
Sep 05 02:26:50.362531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.362461    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxlf2\" (UniqueName: \"kubernetes.io/projected/aa67bfeb-b431-4745-add1-5d03ed865c59-kube-api-access-jxlf2\") pod \"aa67bfeb-b431-4745-add1-5d03ed865c59\" (UID: \"aa67bfeb-b431-4745-add1-5d03ed865c59\") "
Sep 05 02:26:50.362531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.362508    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5gk8\" (UniqueName: \"kubernetes.io/projected/adf46c1f-2692-47b9-a657-ea92346ccc64-kube-api-access-d5gk8\") pod \"adf46c1f-2692-47b9-a657-ea92346ccc64\" (UID: \"adf46c1f-2692-47b9-a657-ea92346ccc64\") "
Sep 05 02:26:50.362839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.362542    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5412088f-ac7e-4d5f-9223-783c5142b4e3-util\") pod \"5412088f-ac7e-4d5f-9223-783c5142b4e3\" (UID: \"5412088f-ac7e-4d5f-9223-783c5142b4e3\") "
Sep 05 02:26:50.362839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.362588    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tsps\" (UniqueName: \"kubernetes.io/projected/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-kube-api-access-8tsps\") pod \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\" (UID: \"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d\") "
Sep 05 02:26:50.362839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.362680    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa67bfeb-b431-4745-add1-5d03ed865c59-bundle\") pod \"aa67bfeb-b431-4745-add1-5d03ed865c59\" (UID: \"aa67bfeb-b431-4745-add1-5d03ed865c59\") "
Sep 05 02:26:50.362839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.362724    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adf46c1f-2692-47b9-a657-ea92346ccc64-bundle\") pod \"adf46c1f-2692-47b9-a657-ea92346ccc64\" (UID: \"adf46c1f-2692-47b9-a657-ea92346ccc64\") "
Sep 05 02:26:50.362839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.362756    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adf46c1f-2692-47b9-a657-ea92346ccc64-util\") pod \"adf46c1f-2692-47b9-a657-ea92346ccc64\" (UID: \"adf46c1f-2692-47b9-a657-ea92346ccc64\") "
Sep 05 02:26:50.363133 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.363103    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-bundle" (OuterVolumeSpecName: "bundle") pod "53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d" (UID: "53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:26:50.363870 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.363832    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5412088f-ac7e-4d5f-9223-783c5142b4e3-bundle" (OuterVolumeSpecName: "bundle") pod "5412088f-ac7e-4d5f-9223-783c5142b4e3" (UID: "5412088f-ac7e-4d5f-9223-783c5142b4e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:26:50.364552 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.364184    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa67bfeb-b431-4745-add1-5d03ed865c59-bundle" (OuterVolumeSpecName: "bundle") pod "aa67bfeb-b431-4745-add1-5d03ed865c59" (UID: "aa67bfeb-b431-4745-add1-5d03ed865c59"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:26:50.366147 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.366112    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adf46c1f-2692-47b9-a657-ea92346ccc64-bundle" (OuterVolumeSpecName: "bundle") pod "adf46c1f-2692-47b9-a657-ea92346ccc64" (UID: "adf46c1f-2692-47b9-a657-ea92346ccc64"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:26:50.366520 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.366492    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa67bfeb-b431-4745-add1-5d03ed865c59-kube-api-access-jxlf2" (OuterVolumeSpecName: "kube-api-access-jxlf2") pod "aa67bfeb-b431-4745-add1-5d03ed865c59" (UID: "aa67bfeb-b431-4745-add1-5d03ed865c59"). InnerVolumeSpecName "kube-api-access-jxlf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:26:50.368911 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.368883    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5412088f-ac7e-4d5f-9223-783c5142b4e3-kube-api-access-stlh6" (OuterVolumeSpecName: "kube-api-access-stlh6") pod "5412088f-ac7e-4d5f-9223-783c5142b4e3" (UID: "5412088f-ac7e-4d5f-9223-783c5142b4e3"). InnerVolumeSpecName "kube-api-access-stlh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:26:50.370360 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.370324    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf46c1f-2692-47b9-a657-ea92346ccc64-kube-api-access-d5gk8" (OuterVolumeSpecName: "kube-api-access-d5gk8") pod "adf46c1f-2692-47b9-a657-ea92346ccc64" (UID: "adf46c1f-2692-47b9-a657-ea92346ccc64"). InnerVolumeSpecName "kube-api-access-d5gk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:26:50.371057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.370698    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-kube-api-access-8tsps" (OuterVolumeSpecName: "kube-api-access-8tsps") pod "53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d" (UID: "53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d"). InnerVolumeSpecName "kube-api-access-8tsps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:26:50.372315 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.372285    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-util" (OuterVolumeSpecName: "util") pod "53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d" (UID: "53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:26:50.373294 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.373264    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa67bfeb-b431-4745-add1-5d03ed865c59-util" (OuterVolumeSpecName: "util") pod "aa67bfeb-b431-4745-add1-5d03ed865c59" (UID: "aa67bfeb-b431-4745-add1-5d03ed865c59"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:26:50.373465 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.373429    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adf46c1f-2692-47b9-a657-ea92346ccc64-util" (OuterVolumeSpecName: "util") pod "adf46c1f-2692-47b9-a657-ea92346ccc64" (UID: "adf46c1f-2692-47b9-a657-ea92346ccc64"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:26:50.374153 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.374132    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5412088f-ac7e-4d5f-9223-783c5142b4e3-util" (OuterVolumeSpecName: "util") pod "5412088f-ac7e-4d5f-9223-783c5142b4e3" (UID: "5412088f-ac7e-4d5f-9223-783c5142b4e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:26:50.464057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.463975    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa67bfeb-b431-4745-add1-5d03ed865c59-bundle\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:50.464057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.464004    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adf46c1f-2692-47b9-a657-ea92346ccc64-bundle\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:50.464057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.464012    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adf46c1f-2692-47b9-a657-ea92346ccc64-util\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:50.464057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.464020    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-bundle\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:50.464057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.464028    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-util\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:50.464057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.464035    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa67bfeb-b431-4745-add1-5d03ed865c59-util\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:50.464057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.464043    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5412088f-ac7e-4d5f-9223-783c5142b4e3-bundle\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:50.464057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.464051    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stlh6\" (UniqueName: \"kubernetes.io/projected/5412088f-ac7e-4d5f-9223-783c5142b4e3-kube-api-access-stlh6\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:50.464057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.464061    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxlf2\" (UniqueName: \"kubernetes.io/projected/aa67bfeb-b431-4745-add1-5d03ed865c59-kube-api-access-jxlf2\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:50.464420 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.464070    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5gk8\" (UniqueName: \"kubernetes.io/projected/adf46c1f-2692-47b9-a657-ea92346ccc64-kube-api-access-d5gk8\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:50.464420 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.464079    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5412088f-ac7e-4d5f-9223-783c5142b4e3-util\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:50.464420 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:50.464088    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tsps\" (UniqueName: \"kubernetes.io/projected/53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d-kube-api-access-8tsps\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:26:51.133052 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:51.133024    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j"
Sep 05 02:26:51.133232 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:51.133018    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537zl4j" event={"ID":"aa67bfeb-b431-4745-add1-5d03ed865c59","Type":"ContainerDied","Data":"61ef585379eafde87e9bcfe5921ad190bc6e915f0177f1fa9de43c4b9c1bf46f"}
Sep 05 02:26:51.133232 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:51.133151    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ef585379eafde87e9bcfe5921ad190bc6e915f0177f1fa9de43c4b9c1bf46f"
Sep 05 02:26:51.135148 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:51.135093    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc"
Sep 05 02:26:51.135288 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:51.135087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0cqkvc" event={"ID":"5412088f-ac7e-4d5f-9223-783c5142b4e3","Type":"ContainerDied","Data":"8d7988d95cf6e0501be339fdfc34ea55e2d17dbcf0548f574a60c7b23d3b31ae"}
Sep 05 02:26:51.135353 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:51.135300    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d7988d95cf6e0501be339fdfc34ea55e2d17dbcf0548f574a60c7b23d3b31ae"
Sep 05 02:26:51.137296 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:51.137275    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7"
Sep 05 02:26:51.137410 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:51.137297    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbpqcc7" event={"ID":"adf46c1f-2692-47b9-a657-ea92346ccc64","Type":"ContainerDied","Data":"f109a03239b603abc7036992653172c8a5a2b6ef0b8b84c5ba5c1287c15ad886"}
Sep 05 02:26:51.137410 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:51.137341    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f109a03239b603abc7036992653172c8a5a2b6ef0b8b84c5ba5c1287c15ad886"
Sep 05 02:26:51.139406 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:51.139382    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm" event={"ID":"53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d","Type":"ContainerDied","Data":"a6aa72852fff40177722d54a2c5ddb125b30b921623dde1a527c378b4316d36f"}
Sep 05 02:26:51.139542 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:51.139422    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6aa72852fff40177722d54a2c5ddb125b30b921623dde1a527c378b4316d36f"
Sep 05 02:26:51.139542 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:51.139421    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183czr8hm"
Sep 05 02:26:54.457626 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:54.457588    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:55.241688 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:55.241656    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:56.040179 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:56.040149    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:26:58.807305 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807266    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj"]
Sep 05 02:26:58.807769 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807714    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5412088f-ac7e-4d5f-9223-783c5142b4e3" containerName="util"
Sep 05 02:26:58.807769 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807735    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5412088f-ac7e-4d5f-9223-783c5142b4e3" containerName="util"
Sep 05 02:26:58.807769 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807749    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adf46c1f-2692-47b9-a657-ea92346ccc64" containerName="util"
Sep 05 02:26:58.807769 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807757    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="adf46c1f-2692-47b9-a657-ea92346ccc64" containerName="util"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807772    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa67bfeb-b431-4745-add1-5d03ed865c59" containerName="util"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807780    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa67bfeb-b431-4745-add1-5d03ed865c59" containerName="util"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807789    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adf46c1f-2692-47b9-a657-ea92346ccc64" containerName="pull"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807798    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="adf46c1f-2692-47b9-a657-ea92346ccc64" containerName="pull"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807807    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa67bfeb-b431-4745-add1-5d03ed865c59" containerName="pull"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807814    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa67bfeb-b431-4745-add1-5d03ed865c59" containerName="pull"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807823    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5412088f-ac7e-4d5f-9223-783c5142b4e3" containerName="extract"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807832    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5412088f-ac7e-4d5f-9223-783c5142b4e3" containerName="extract"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807847    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adf46c1f-2692-47b9-a657-ea92346ccc64" containerName="extract"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807855    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="adf46c1f-2692-47b9-a657-ea92346ccc64" containerName="extract"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807868    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d" containerName="util"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807875    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d" containerName="util"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807885    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5412088f-ac7e-4d5f-9223-783c5142b4e3" containerName="pull"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807892    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5412088f-ac7e-4d5f-9223-783c5142b4e3" containerName="pull"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807903    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa67bfeb-b431-4745-add1-5d03ed865c59" containerName="extract"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807912    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa67bfeb-b431-4745-add1-5d03ed865c59" containerName="extract"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807930    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d" containerName="extract"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807938    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d" containerName="extract"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807949    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d" containerName="pull"
Sep 05 02:26:58.807988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.807958    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d" containerName="pull"
Sep 05 02:26:58.808926 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.808038    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5412088f-ac7e-4d5f-9223-783c5142b4e3" containerName="extract"
Sep 05 02:26:58.808926 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.808051    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="53b7fe46-ff99-45d6-a4a6-a7ec52c3d79d" containerName="extract"
Sep 05 02:26:58.808926 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.808062    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa67bfeb-b431-4745-add1-5d03ed865c59" containerName="extract"
Sep 05 02:26:58.808926 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.808072    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="adf46c1f-2692-47b9-a657-ea92346ccc64" containerName="extract"
Sep 05 02:26:58.812629 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.812607    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj"
Sep 05 02:26:58.814801 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.814779    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-g2cv9\""
Sep 05 02:26:58.814801 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.814793    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 05 02:26:58.814978 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.814839    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 05 02:26:58.824134 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.824112    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj"]
Sep 05 02:26:58.935308 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.935274    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3159908c-3142-42b9-b97d-590ea1610462-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-9cmdj\" (UID: \"3159908c-3142-42b9-b97d-590ea1610462\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj"
Sep 05 02:26:58.935497 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:58.935319    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l6vr\" (UniqueName: \"kubernetes.io/projected/3159908c-3142-42b9-b97d-590ea1610462-kube-api-access-5l6vr\") pod \"kuadrant-operator-controller-manager-66866f5fcc-9cmdj\" (UID: \"3159908c-3142-42b9-b97d-590ea1610462\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj"
Sep 05 02:26:59.035886 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:59.035845    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l6vr\" (UniqueName: \"kubernetes.io/projected/3159908c-3142-42b9-b97d-590ea1610462-kube-api-access-5l6vr\") pod \"kuadrant-operator-controller-manager-66866f5fcc-9cmdj\" (UID: \"3159908c-3142-42b9-b97d-590ea1610462\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj"
Sep 05 02:26:59.036067 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:59.035975    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3159908c-3142-42b9-b97d-590ea1610462-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-9cmdj\" (UID: \"3159908c-3142-42b9-b97d-590ea1610462\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj"
Sep 05 02:26:59.036330 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:59.036307    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3159908c-3142-42b9-b97d-590ea1610462-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-9cmdj\" (UID: \"3159908c-3142-42b9-b97d-590ea1610462\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj"
Sep 05 02:26:59.043884 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:59.043861    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l6vr\" (UniqueName: \"kubernetes.io/projected/3159908c-3142-42b9-b97d-590ea1610462-kube-api-access-5l6vr\") pod \"kuadrant-operator-controller-manager-66866f5fcc-9cmdj\" (UID: \"3159908c-3142-42b9-b97d-590ea1610462\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj"
Sep 05 02:26:59.122562 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:59.122526    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj"
Sep 05 02:26:59.399592 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:59.399557    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj"]
Sep 05 02:26:59.401276 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:26:59.401235    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3159908c_3142_42b9_b97d_590ea1610462.slice/crio-fa6f4f94f4b3dc8da9f0f82d1006ca35d01d4c5642122a4e472c03cd1aa0ac56 WatchSource:0}: Error finding container fa6f4f94f4b3dc8da9f0f82d1006ca35d01d4c5642122a4e472c03cd1aa0ac56: Status 404 returned error can't find the container with id fa6f4f94f4b3dc8da9f0f82d1006ca35d01d4c5642122a4e472c03cd1aa0ac56
Sep 05 02:26:59.493596 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:26:59.493549    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:00.158964 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:00.158909    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:00.191253 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:00.191211    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj" event={"ID":"3159908c-3142-42b9-b97d-590ea1610462","Type":"ContainerStarted","Data":"fa6f4f94f4b3dc8da9f0f82d1006ca35d01d4c5642122a4e472c03cd1aa0ac56"}
Sep 05 02:27:01.461567 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:01.461528    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:02.489828 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:02.489788    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:02.873525 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:02.873493    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-czc6z"]
Sep 05 02:27:02.876698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:02.876671    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-czc6z"
Sep 05 02:27:02.879347 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:02.879137    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5hjxr\""
Sep 05 02:27:02.889495 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:02.889471    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-czc6z"]
Sep 05 02:27:02.973873 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:02.973837    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69gh6\" (UniqueName: \"kubernetes.io/projected/679a80f2-3028-4d51-a687-e2d5d7a36341-kube-api-access-69gh6\") pod \"limitador-operator-controller-manager-774d94476f-czc6z\" (UID: \"679a80f2-3028-4d51-a687-e2d5d7a36341\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-czc6z"
Sep 05 02:27:03.074644 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:03.074600    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gh6\" (UniqueName: \"kubernetes.io/projected/679a80f2-3028-4d51-a687-e2d5d7a36341-kube-api-access-69gh6\") pod \"limitador-operator-controller-manager-774d94476f-czc6z\" (UID: \"679a80f2-3028-4d51-a687-e2d5d7a36341\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-czc6z"
Sep 05 02:27:03.090413 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:03.090386    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gh6\" (UniqueName: \"kubernetes.io/projected/679a80f2-3028-4d51-a687-e2d5d7a36341-kube-api-access-69gh6\") pod \"limitador-operator-controller-manager-774d94476f-czc6z\" (UID: \"679a80f2-3028-4d51-a687-e2d5d7a36341\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-czc6z"
Sep 05 02:27:03.190682 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:03.190607    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-czc6z"
Sep 05 02:27:03.248696 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:03.248651    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:03.655537 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:03.655492    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:03.898938 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:03.898902    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:04.645462 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:04.645418    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-czc6z"]
Sep 05 02:27:04.646991 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:27:04.646953    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679a80f2_3028_4d51_a687_e2d5d7a36341.slice/crio-bdf6ce817f8f29130757173a8c9851ea3cd15fa843f7b94254f83bd93b85e7c8 WatchSource:0}: Error finding container bdf6ce817f8f29130757173a8c9851ea3cd15fa843f7b94254f83bd93b85e7c8: Status 404 returned error can't find the container with id bdf6ce817f8f29130757173a8c9851ea3cd15fa843f7b94254f83bd93b85e7c8
Sep 05 02:27:05.228385 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:05.228344    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-czc6z" event={"ID":"679a80f2-3028-4d51-a687-e2d5d7a36341","Type":"ContainerStarted","Data":"bdf6ce817f8f29130757173a8c9851ea3cd15fa843f7b94254f83bd93b85e7c8"}
Sep 05 02:27:05.230106 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:05.230078    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj" event={"ID":"3159908c-3142-42b9-b97d-590ea1610462","Type":"ContainerStarted","Data":"5807ea3f84760de049056ed675e17ef4e21fc1cc4f16bcb7c39d833a9e420394"}
Sep 05 02:27:05.230243 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:05.230223    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj"
Sep 05 02:27:05.256717 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:05.256673    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj" podStartSLOduration=2.19963642 podStartE2EDuration="7.256658288s" totalImagesPullingTime="5.057021868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:27:05.256068697 +0000 UTC m=+528.811225006" watchObservedRunningTime="2026-09-05 02:27:05.256658288 +0000 UTC m=+528.811814598"
Sep 05 02:27:06.139731 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:06.139698    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:07.113557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:07.113521    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:07.242497 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:07.242455    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-czc6z" event={"ID":"679a80f2-3028-4d51-a687-e2d5d7a36341","Type":"ContainerStarted","Data":"23853d543434fd7618f5a29dacdea89c2331f8366d96cec55a0e30ccf13d3bdf"}
Sep 05 02:27:07.242711 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:07.242579    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-czc6z"
Sep 05 02:27:07.260945 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:07.260894    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-czc6z" podStartSLOduration=3.457748836 podStartE2EDuration="5.260877942s" totalImagesPullingTime="1.803129106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:27:07.259502097 +0000 UTC m=+530.814658407" watchObservedRunningTime="2026-09-05 02:27:07.260877942 +0000 UTC m=+530.816034250"
Sep 05 02:27:07.433719 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:07.433627    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:09.647595 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:09.647500    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:09.703968 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:09.703934    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:10.247260 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:10.247226    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:10.299937 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:10.299903    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:10.668954 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:10.668919    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:10.800378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:10.800336    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:16.238712 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:16.238677    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-9cmdj"
Sep 05 02:27:16.538948 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:16.538861    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:16.640170 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:16.640131    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:18.250370 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:18.250337    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-czc6z"
Sep 05 02:27:21.283033 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:21.282963    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:21.390758 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:21.390717    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:26.084123 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:26.084084    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:26.183365 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:26.183332    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:30.921707 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:30.921669    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:31.049571 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:31.049538    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:31.149524 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:31.149485    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:37.435303 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:37.435269    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:37.537556 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:37.537506    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:41.740414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:41.740376    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:41.840055 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:41.840014    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:42.174740 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:42.174691    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:42.276413 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:42.276378    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:46.988175 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:46.988134    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:47.103100 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:47.103056    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:50.579059 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:50.579027    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:50.689397 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:50.689359    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:50.758491 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:50.758452    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:50.998482 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:50.998381    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6sxln"]
Sep 05 02:27:51.001538 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.001502    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-6sxln"
Sep 05 02:27:51.005905 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.005882    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 05 02:27:51.006202 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.006180    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w4nr9\""
Sep 05 02:27:51.013419 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.013390    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6sxln"]
Sep 05 02:27:51.046997 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.046964    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6sxln"]
Sep 05 02:27:51.076106 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.076078    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31b61bd5-b792-4925-a4ff-94ccbf297f78-config-file\") pod \"limitador-limitador-85d9db4d5d-6sxln\" (UID: \"31b61bd5-b792-4925-a4ff-94ccbf297f78\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-6sxln"
Sep 05 02:27:51.076256 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.076178    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvf26\" (UniqueName: \"kubernetes.io/projected/31b61bd5-b792-4925-a4ff-94ccbf297f78-kube-api-access-wvf26\") pod \"limitador-limitador-85d9db4d5d-6sxln\" (UID: \"31b61bd5-b792-4925-a4ff-94ccbf297f78\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-6sxln"
Sep 05 02:27:51.177716 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.177669    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31b61bd5-b792-4925-a4ff-94ccbf297f78-config-file\") pod \"limitador-limitador-85d9db4d5d-6sxln\" (UID: \"31b61bd5-b792-4925-a4ff-94ccbf297f78\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-6sxln"
Sep 05 02:27:51.177926 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.177898    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvf26\" (UniqueName: \"kubernetes.io/projected/31b61bd5-b792-4925-a4ff-94ccbf297f78-kube-api-access-wvf26\") pod \"limitador-limitador-85d9db4d5d-6sxln\" (UID: \"31b61bd5-b792-4925-a4ff-94ccbf297f78\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-6sxln"
Sep 05 02:27:51.178329 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.178309    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31b61bd5-b792-4925-a4ff-94ccbf297f78-config-file\") pod \"limitador-limitador-85d9db4d5d-6sxln\" (UID: \"31b61bd5-b792-4925-a4ff-94ccbf297f78\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-6sxln"
Sep 05 02:27:51.186682 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.186655    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvf26\" (UniqueName: \"kubernetes.io/projected/31b61bd5-b792-4925-a4ff-94ccbf297f78-kube-api-access-wvf26\") pod \"limitador-limitador-85d9db4d5d-6sxln\" (UID: \"31b61bd5-b792-4925-a4ff-94ccbf297f78\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-6sxln"
Sep 05 02:27:51.314743 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.314659    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-6sxln"
Sep 05 02:27:51.433122 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.433085    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:51.532077 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.532044    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:51.587621 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:51.587583    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-6sxln"]
Sep 05 02:27:51.590498 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:27:51.590471    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b61bd5_b792_4925_a4ff_94ccbf297f78.slice/crio-6561de4d4a40281ea3dc67662978b688058d281923cd5516421a9a45b472c00e WatchSource:0}: Error finding container 6561de4d4a40281ea3dc67662978b688058d281923cd5516421a9a45b472c00e: Status 404 returned error can't find the container with id 6561de4d4a40281ea3dc67662978b688058d281923cd5516421a9a45b472c00e
Sep 05 02:27:52.186321 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:52.186267    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:52.251657 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:52.251617    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:52.324285 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:52.324252    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:52.466248 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:52.466168    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-6sxln" event={"ID":"31b61bd5-b792-4925-a4ff-94ccbf297f78","Type":"ContainerStarted","Data":"6561de4d4a40281ea3dc67662978b688058d281923cd5516421a9a45b472c00e"}
Sep 05 02:27:52.588537 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:52.588500    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:52.689455 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:52.689415    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:52.789720 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:52.789630    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:52.890172 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:52.890140    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:54.478306 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:54.478274    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-6sxln" event={"ID":"31b61bd5-b792-4925-a4ff-94ccbf297f78","Type":"ContainerStarted","Data":"5a51e9f60058f711fa622d98295fe94753e80d8859b798eeda6b007db08f9eee"}
Sep 05 02:27:54.478720 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:54.478340    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-6sxln"
Sep 05 02:27:54.496271 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:54.496218    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-6sxln" podStartSLOduration=2.529273736 podStartE2EDuration="4.496201451s" totalImagesPullingTime="1.966927715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:27:54.494343462 +0000 UTC m=+578.049499774" watchObservedRunningTime="2026-09-05 02:27:54.496201451 +0000 UTC m=+578.051357777"
Sep 05 02:27:58.850998 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:58.850965    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:27:58.951506 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:27:58.951466    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:03.633708 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:03.633661    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:03.734835 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:03.734798    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:05.484151 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:05.484122    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-6sxln"
Sep 05 02:28:08.392665 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:08.392630    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:08.492386 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:08.492349    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:10.639335 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:10.639296    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:10.739114 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:10.739080    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:11.618777 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:11.618740    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:11.761541 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:11.761504    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:16.332917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:16.332883    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:16.865275 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:16.865248    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:28:16.865870 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:16.865846    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:28:19.741422 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:19.741370    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:19.843184 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:19.843153    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:39.368592 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.368515    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"]
Sep 05 02:28:39.372246 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.368711    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds" podUID="30f09e1f-2bf1-4837-891f-cab2ea128a2a" containerName="discovery" containerID="cri-o://49822b4f17de37a5e0babc81d5313b764c18ca922018816bc7dca729df49846d" gracePeriod=30
Sep 05 02:28:39.613218 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.613192    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:28:39.666875 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.666810    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-kubeconfig\") pod \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") "
Sep 05 02:28:39.666986 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.666872    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-cacerts\") pod \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") "
Sep 05 02:28:39.666986 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.666906    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-csr-ca-configmap\") pod \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") "
Sep 05 02:28:39.666986 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.666958    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-csr-dns-cert\") pod \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") "
Sep 05 02:28:39.667137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.666990    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f09e1f-2bf1-4837-891f-cab2ea128a2a-local-certs\") pod \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") "
Sep 05 02:28:39.667137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.667034    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wbms\" (UniqueName: \"kubernetes.io/projected/30f09e1f-2bf1-4837-891f-cab2ea128a2a-kube-api-access-5wbms\") pod \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") "
Sep 05 02:28:39.667137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.667088    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-token\") pod \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\" (UID: \"30f09e1f-2bf1-4837-891f-cab2ea128a2a\") "
Sep 05 02:28:39.667421 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.667365    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "30f09e1f-2bf1-4837-891f-cab2ea128a2a" (UID: "30f09e1f-2bf1-4837-891f-cab2ea128a2a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:28:39.669218 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.669172    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "30f09e1f-2bf1-4837-891f-cab2ea128a2a" (UID: "30f09e1f-2bf1-4837-891f-cab2ea128a2a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:28:39.669218 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.669196    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-cacerts" (OuterVolumeSpecName: "cacerts") pod "30f09e1f-2bf1-4837-891f-cab2ea128a2a" (UID: "30f09e1f-2bf1-4837-891f-cab2ea128a2a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:28:39.669396 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.669374    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "30f09e1f-2bf1-4837-891f-cab2ea128a2a" (UID: "30f09e1f-2bf1-4837-891f-cab2ea128a2a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:28:39.669455 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.669390    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f09e1f-2bf1-4837-891f-cab2ea128a2a-kube-api-access-5wbms" (OuterVolumeSpecName: "kube-api-access-5wbms") pod "30f09e1f-2bf1-4837-891f-cab2ea128a2a" (UID: "30f09e1f-2bf1-4837-891f-cab2ea128a2a"). InnerVolumeSpecName "kube-api-access-5wbms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:28:39.669627 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.669606    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-token" (OuterVolumeSpecName: "istio-token") pod "30f09e1f-2bf1-4837-891f-cab2ea128a2a" (UID: "30f09e1f-2bf1-4837-891f-cab2ea128a2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:28:39.669680 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.669652    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f09e1f-2bf1-4837-891f-cab2ea128a2a-local-certs" (OuterVolumeSpecName: "local-certs") pod "30f09e1f-2bf1-4837-891f-cab2ea128a2a" (UID: "30f09e1f-2bf1-4837-891f-cab2ea128a2a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:28:39.689878 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.689850    2557 generic.go:356] "Generic (PLEG): container finished" podID="30f09e1f-2bf1-4837-891f-cab2ea128a2a" containerID="49822b4f17de37a5e0babc81d5313b764c18ca922018816bc7dca729df49846d" exitCode=0
Sep 05 02:28:39.689945 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.689895    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"
Sep 05 02:28:39.689945 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.689927    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds" event={"ID":"30f09e1f-2bf1-4837-891f-cab2ea128a2a","Type":"ContainerDied","Data":"49822b4f17de37a5e0babc81d5313b764c18ca922018816bc7dca729df49846d"}
Sep 05 02:28:39.690017 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.689983    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds" event={"ID":"30f09e1f-2bf1-4837-891f-cab2ea128a2a","Type":"ContainerDied","Data":"b02aa0deec1469eb65548b6522c09a4875f5df2bce840682b375d10ffcea213b"}
Sep 05 02:28:39.690017 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.689998    2557 scope.go:117] "RemoveContainer" containerID="49822b4f17de37a5e0babc81d5313b764c18ca922018816bc7dca729df49846d"
Sep 05 02:28:39.698368 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.698352    2557 scope.go:117] "RemoveContainer" containerID="49822b4f17de37a5e0babc81d5313b764c18ca922018816bc7dca729df49846d"
Sep 05 02:28:39.698639 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:28:39.698620    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49822b4f17de37a5e0babc81d5313b764c18ca922018816bc7dca729df49846d\": container with ID starting with 49822b4f17de37a5e0babc81d5313b764c18ca922018816bc7dca729df49846d not found: ID does not exist" containerID="49822b4f17de37a5e0babc81d5313b764c18ca922018816bc7dca729df49846d"
Sep 05 02:28:39.698703 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.698647    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49822b4f17de37a5e0babc81d5313b764c18ca922018816bc7dca729df49846d"} err="failed to get container status \"49822b4f17de37a5e0babc81d5313b764c18ca922018816bc7dca729df49846d\": rpc error: code = NotFound desc = could not find container \"49822b4f17de37a5e0babc81d5313b764c18ca922018816bc7dca729df49846d\": container with ID starting with 49822b4f17de37a5e0babc81d5313b764c18ca922018816bc7dca729df49846d not found: ID does not exist"
Sep 05 02:28:39.711165 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.711142    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"]
Sep 05 02:28:39.714808 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.714786    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kj7ds"]
Sep 05 02:28:39.767961 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.767939    2557 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-cacerts\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:28:39.767961 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.767959    2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-csr-ca-configmap\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:28:39.768068 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.767970    2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-csr-dns-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:28:39.768068 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.767979    2557 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f09e1f-2bf1-4837-891f-cab2ea128a2a-local-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:28:39.768068 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.767988    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wbms\" (UniqueName: \"kubernetes.io/projected/30f09e1f-2bf1-4837-891f-cab2ea128a2a-kube-api-access-5wbms\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:28:39.768068 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.767996    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-token\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:28:39.768068 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:39.768003    2557 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30f09e1f-2bf1-4837-891f-cab2ea128a2a-istio-kubeconfig\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:28:40.989585 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:40.989556    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30f09e1f-2bf1-4837-891f-cab2ea128a2a" path="/var/lib/kubelet/pods/30f09e1f-2bf1-4837-891f-cab2ea128a2a/volumes"
Sep 05 02:28:47.387231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.387196    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:47.447462 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.447419    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-khtsn"]
Sep 05 02:28:47.447797 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.447784    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30f09e1f-2bf1-4837-891f-cab2ea128a2a" containerName="discovery"
Sep 05 02:28:47.447998 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.447799    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="30f09e1f-2bf1-4837-891f-cab2ea128a2a" containerName="discovery"
Sep 05 02:28:47.447998 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.447870    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="30f09e1f-2bf1-4837-891f-cab2ea128a2a" containerName="discovery"
Sep 05 02:28:47.450892 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.450872    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn"
Sep 05 02:28:47.453231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.453211    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 05 02:28:47.453310 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.453218    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 05 02:28:47.453924 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.453910    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-sqb84\""
Sep 05 02:28:47.453997 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.453917    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 05 02:28:47.461152 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.461131    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-khtsn"]
Sep 05 02:28:47.463821 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.463795    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk"]
Sep 05 02:28:47.466882 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.466868    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk"
Sep 05 02:28:47.468748 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.468730    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 05 02:28:47.468983 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.468965    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-gzwrf\""
Sep 05 02:28:47.476647 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.476623    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk"]
Sep 05 02:28:47.488375 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.488347    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-zdgtr"]
Sep 05 02:28:47.491685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.491661    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-zdgtr"
Sep 05 02:28:47.494040 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.493896    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-f5smr\""
Sep 05 02:28:47.494262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.493905    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 05 02:28:47.502429 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.502404    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-zdgtr"]
Sep 05 02:28:47.512648 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.512621    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:47.527488 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.527462    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldh8l\" (UniqueName: \"kubernetes.io/projected/bc5ad237-ad5c-4817-ba9a-148cfd194e99-kube-api-access-ldh8l\") pod \"seaweedfs-64568bcd49-zdgtr\" (UID: \"bc5ad237-ad5c-4817-ba9a-148cfd194e99\") " pod="kserve/seaweedfs-64568bcd49-zdgtr"
Sep 05 02:28:47.527586 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.527534    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nbqb\" (UniqueName: \"kubernetes.io/projected/db252401-bd9d-4cdc-b400-3e07d2ae2377-kube-api-access-5nbqb\") pod \"kserve-controller-manager-577b5fb6f8-khtsn\" (UID: \"db252401-bd9d-4cdc-b400-3e07d2ae2377\") " pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn"
Sep 05 02:28:47.527586 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.527578    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qz95\" (UniqueName: \"kubernetes.io/projected/0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82-kube-api-access-8qz95\") pod \"llmisvc-controller-manager-7bcc44dcb6-nndsk\" (UID: \"0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82\") " pod="kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk"
Sep 05 02:28:47.527674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.527610    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db252401-bd9d-4cdc-b400-3e07d2ae2377-cert\") pod \"kserve-controller-manager-577b5fb6f8-khtsn\" (UID: \"db252401-bd9d-4cdc-b400-3e07d2ae2377\") " pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn"
Sep 05 02:28:47.527674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.527634    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82-cert\") pod \"llmisvc-controller-manager-7bcc44dcb6-nndsk\" (UID: \"0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82\") " pod="kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk"
Sep 05 02:28:47.527674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.527662    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc5ad237-ad5c-4817-ba9a-148cfd194e99-data\") pod \"seaweedfs-64568bcd49-zdgtr\" (UID: \"bc5ad237-ad5c-4817-ba9a-148cfd194e99\") " pod="kserve/seaweedfs-64568bcd49-zdgtr"
Sep 05 02:28:47.606639 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.606603    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:47.628251 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.628221    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldh8l\" (UniqueName: \"kubernetes.io/projected/bc5ad237-ad5c-4817-ba9a-148cfd194e99-kube-api-access-ldh8l\") pod \"seaweedfs-64568bcd49-zdgtr\" (UID: \"bc5ad237-ad5c-4817-ba9a-148cfd194e99\") " pod="kserve/seaweedfs-64568bcd49-zdgtr"
Sep 05 02:28:47.628347 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.628267    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nbqb\" (UniqueName: \"kubernetes.io/projected/db252401-bd9d-4cdc-b400-3e07d2ae2377-kube-api-access-5nbqb\") pod \"kserve-controller-manager-577b5fb6f8-khtsn\" (UID: \"db252401-bd9d-4cdc-b400-3e07d2ae2377\") " pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn"
Sep 05 02:28:47.628347 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.628307    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qz95\" (UniqueName: \"kubernetes.io/projected/0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82-kube-api-access-8qz95\") pod \"llmisvc-controller-manager-7bcc44dcb6-nndsk\" (UID: \"0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82\") " pod="kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk"
Sep 05 02:28:47.628347 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.628336    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db252401-bd9d-4cdc-b400-3e07d2ae2377-cert\") pod \"kserve-controller-manager-577b5fb6f8-khtsn\" (UID: \"db252401-bd9d-4cdc-b400-3e07d2ae2377\") " pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn"
Sep 05 02:28:47.628530 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.628357    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82-cert\") pod \"llmisvc-controller-manager-7bcc44dcb6-nndsk\" (UID: \"0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82\") " pod="kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk"
Sep 05 02:28:47.628530 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.628381    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc5ad237-ad5c-4817-ba9a-148cfd194e99-data\") pod \"seaweedfs-64568bcd49-zdgtr\" (UID: \"bc5ad237-ad5c-4817-ba9a-148cfd194e99\") " pod="kserve/seaweedfs-64568bcd49-zdgtr"
Sep 05 02:28:47.628530 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:28:47.628477    2557 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 05 02:28:47.628530 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:28:47.628489    2557 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 05 02:28:47.628698 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:28:47.628556    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db252401-bd9d-4cdc-b400-3e07d2ae2377-cert podName:db252401-bd9d-4cdc-b400-3e07d2ae2377 nodeName:}" failed. No retries permitted until 2026-09-05 02:28:48.128536926 +0000 UTC m=+631.683693220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/db252401-bd9d-4cdc-b400-3e07d2ae2377-cert") pod "kserve-controller-manager-577b5fb6f8-khtsn" (UID: "db252401-bd9d-4cdc-b400-3e07d2ae2377") : secret "kserve-webhook-server-cert" not found
Sep 05 02:28:47.628698 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:28:47.628575    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82-cert podName:0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82 nodeName:}" failed. No retries permitted until 2026-09-05 02:28:48.128565655 +0000 UTC m=+631.683721955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82-cert") pod "llmisvc-controller-manager-7bcc44dcb6-nndsk" (UID: "0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82") : secret "llmisvc-webhook-server-cert" not found
Sep 05 02:28:47.628800 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.628782    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc5ad237-ad5c-4817-ba9a-148cfd194e99-data\") pod \"seaweedfs-64568bcd49-zdgtr\" (UID: \"bc5ad237-ad5c-4817-ba9a-148cfd194e99\") " pod="kserve/seaweedfs-64568bcd49-zdgtr"
Sep 05 02:28:47.637796 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.637732    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nbqb\" (UniqueName: \"kubernetes.io/projected/db252401-bd9d-4cdc-b400-3e07d2ae2377-kube-api-access-5nbqb\") pod \"kserve-controller-manager-577b5fb6f8-khtsn\" (UID: \"db252401-bd9d-4cdc-b400-3e07d2ae2377\") " pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn"
Sep 05 02:28:47.638110 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.638086    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qz95\" (UniqueName: \"kubernetes.io/projected/0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82-kube-api-access-8qz95\") pod \"llmisvc-controller-manager-7bcc44dcb6-nndsk\" (UID: \"0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82\") " pod="kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk"
Sep 05 02:28:47.638229 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.638210    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldh8l\" (UniqueName: \"kubernetes.io/projected/bc5ad237-ad5c-4817-ba9a-148cfd194e99-kube-api-access-ldh8l\") pod \"seaweedfs-64568bcd49-zdgtr\" (UID: \"bc5ad237-ad5c-4817-ba9a-148cfd194e99\") " pod="kserve/seaweedfs-64568bcd49-zdgtr"
Sep 05 02:28:47.781760 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.781724    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:47.801754 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.801735    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-zdgtr"
Sep 05 02:28:47.880992 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:47.880956    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:48.071323 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.071295    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-zdgtr"]
Sep 05 02:28:48.133606 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.133560    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db252401-bd9d-4cdc-b400-3e07d2ae2377-cert\") pod \"kserve-controller-manager-577b5fb6f8-khtsn\" (UID: \"db252401-bd9d-4cdc-b400-3e07d2ae2377\") " pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn"
Sep 05 02:28:48.133606 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.133608    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82-cert\") pod \"llmisvc-controller-manager-7bcc44dcb6-nndsk\" (UID: \"0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82\") " pod="kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk"
Sep 05 02:28:48.136092 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.136064    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82-cert\") pod \"llmisvc-controller-manager-7bcc44dcb6-nndsk\" (UID: \"0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82\") " pod="kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk"
Sep 05 02:28:48.136188 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.136066    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db252401-bd9d-4cdc-b400-3e07d2ae2377-cert\") pod \"kserve-controller-manager-577b5fb6f8-khtsn\" (UID: \"db252401-bd9d-4cdc-b400-3e07d2ae2377\") " pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn"
Sep 05 02:28:48.361475 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.361361    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn"
Sep 05 02:28:48.377385 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.377363    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk"
Sep 05 02:28:48.420318 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.420218    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:48.516648 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.516616    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:48.645294 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.645244    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-khtsn"]
Sep 05 02:28:48.645878 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.645849    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:28:48.647521 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:28:48.647495    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb252401_bd9d_4cdc_b400_3e07d2ae2377.slice/crio-4d89f3cb75d543b2d4c25693aaf18840dd011ac7ebd834a822dea7d9b5cd9b5b WatchSource:0}: Error finding container 4d89f3cb75d543b2d4c25693aaf18840dd011ac7ebd834a822dea7d9b5cd9b5b: Status 404 returned error can't find the container with id 4d89f3cb75d543b2d4c25693aaf18840dd011ac7ebd834a822dea7d9b5cd9b5b
Sep 05 02:28:48.680410 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.680386    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk"]
Sep 05 02:28:48.682149 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:28:48.682122    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0bfb153b_7f0e_4e59_80d8_4f7bd5bbae82.slice/crio-fbb80bacce83da8d84c646d922af6fbfeb18de5468d77383f4bd096cb77c652f WatchSource:0}: Error finding container fbb80bacce83da8d84c646d922af6fbfeb18de5468d77383f4bd096cb77c652f: Status 404 returned error can't find the container with id fbb80bacce83da8d84c646d922af6fbfeb18de5468d77383f4bd096cb77c652f
Sep 05 02:28:48.738508 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.738418    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn" event={"ID":"db252401-bd9d-4cdc-b400-3e07d2ae2377","Type":"ContainerStarted","Data":"4d89f3cb75d543b2d4c25693aaf18840dd011ac7ebd834a822dea7d9b5cd9b5b"}
Sep 05 02:28:48.740146 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.740115    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-zdgtr" event={"ID":"bc5ad237-ad5c-4817-ba9a-148cfd194e99","Type":"ContainerStarted","Data":"adf726cfbe584f861c84b6de3c878da1e435908eb0309587c0a39e8119ece3f1"}
Sep 05 02:28:48.741668 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:48.741639    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk" event={"ID":"0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82","Type":"ContainerStarted","Data":"fbb80bacce83da8d84c646d922af6fbfeb18de5468d77383f4bd096cb77c652f"}
Sep 05 02:28:52.765265 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:52.765226    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk" event={"ID":"0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82","Type":"ContainerStarted","Data":"c39a0373322121d087f770be756f492180e722650d2377f4bcc8ff573aa2e0e3"}
Sep 05 02:28:52.765743 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:52.765295    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk"
Sep 05 02:28:52.766908 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:52.766883    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn" event={"ID":"db252401-bd9d-4cdc-b400-3e07d2ae2377","Type":"ContainerStarted","Data":"0ed04e50ec43d92102bf99d477084c6393b2bb17eb62a1bff148609e49e85116"}
Sep 05 02:28:52.767022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:52.766932    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn"
Sep 05 02:28:52.768414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:52.768393    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-zdgtr" event={"ID":"bc5ad237-ad5c-4817-ba9a-148cfd194e99","Type":"ContainerStarted","Data":"c631962742220986b6c854d8c9a37ab2d53d1cbd7d153effeebee03f2e6d9bf7"}
Sep 05 02:28:52.768530 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:52.768488    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-zdgtr"
Sep 05 02:28:52.784210 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:52.784153    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk" podStartSLOduration=2.412082423 podStartE2EDuration="5.784137352s" totalImagesPullingTime="3.372054929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:28:52.783500127 +0000 UTC m=+636.338656437" watchObservedRunningTime="2026-09-05 02:28:52.784137352 +0000 UTC m=+636.339293660"
Sep 05 02:28:52.801520 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:52.801476    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-zdgtr" podStartSLOduration=1.693655372 podStartE2EDuration="5.801462394s" totalImagesPullingTime="4.107807022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:28:52.800153303 +0000 UTC m=+636.355309611" watchObservedRunningTime="2026-09-05 02:28:52.801462394 +0000 UTC m=+636.356618695"
Sep 05 02:28:52.818455 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:52.818395    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn" podStartSLOduration=2.373872031 podStartE2EDuration="5.818377655s" totalImagesPullingTime="3.444505624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:28:52.816396572 +0000 UTC m=+636.371552879" watchObservedRunningTime="2026-09-05 02:28:52.818377655 +0000 UTC m=+636.373533965"
Sep 05 02:28:58.776285 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:28:58.776251    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-zdgtr"
Sep 05 02:29:23.776093 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:23.776061    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-7bcc44dcb6-nndsk"
Sep 05 02:29:23.778749 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:23.778729    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn"
Sep 05 02:29:31.510704 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.510672    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-khtsn"]
Sep 05 02:29:31.511080 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.510852    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn" podUID="db252401-bd9d-4cdc-b400-3e07d2ae2377" containerName="manager" containerID="cri-o://0ed04e50ec43d92102bf99d477084c6393b2bb17eb62a1bff148609e49e85116" gracePeriod=10
Sep 05 02:29:31.535629 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.535600    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-tjpj4"]
Sep 05 02:29:31.538167 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.538150    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-tjpj4"
Sep 05 02:29:31.547911 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.547888    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-tjpj4"]
Sep 05 02:29:31.601947 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.601920    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpv6n\" (UniqueName: \"kubernetes.io/projected/44a6af99-2320-483c-af78-37c506467fc4-kube-api-access-rpv6n\") pod \"kserve-controller-manager-577b5fb6f8-tjpj4\" (UID: \"44a6af99-2320-483c-af78-37c506467fc4\") " pod="kserve/kserve-controller-manager-577b5fb6f8-tjpj4"
Sep 05 02:29:31.602078 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.601992    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44a6af99-2320-483c-af78-37c506467fc4-cert\") pod \"kserve-controller-manager-577b5fb6f8-tjpj4\" (UID: \"44a6af99-2320-483c-af78-37c506467fc4\") " pod="kserve/kserve-controller-manager-577b5fb6f8-tjpj4"
Sep 05 02:29:31.703207 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.703176    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpv6n\" (UniqueName: \"kubernetes.io/projected/44a6af99-2320-483c-af78-37c506467fc4-kube-api-access-rpv6n\") pod \"kserve-controller-manager-577b5fb6f8-tjpj4\" (UID: \"44a6af99-2320-483c-af78-37c506467fc4\") " pod="kserve/kserve-controller-manager-577b5fb6f8-tjpj4"
Sep 05 02:29:31.703363 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.703245    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44a6af99-2320-483c-af78-37c506467fc4-cert\") pod \"kserve-controller-manager-577b5fb6f8-tjpj4\" (UID: \"44a6af99-2320-483c-af78-37c506467fc4\") " pod="kserve/kserve-controller-manager-577b5fb6f8-tjpj4"
Sep 05 02:29:31.705380 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.705356    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44a6af99-2320-483c-af78-37c506467fc4-cert\") pod \"kserve-controller-manager-577b5fb6f8-tjpj4\" (UID: \"44a6af99-2320-483c-af78-37c506467fc4\") " pod="kserve/kserve-controller-manager-577b5fb6f8-tjpj4"
Sep 05 02:29:31.712056 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.712031    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpv6n\" (UniqueName: \"kubernetes.io/projected/44a6af99-2320-483c-af78-37c506467fc4-kube-api-access-rpv6n\") pod \"kserve-controller-manager-577b5fb6f8-tjpj4\" (UID: \"44a6af99-2320-483c-af78-37c506467fc4\") " pod="kserve/kserve-controller-manager-577b5fb6f8-tjpj4"
Sep 05 02:29:31.739505 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.739487    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn"
Sep 05 02:29:31.804424 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.804361    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nbqb\" (UniqueName: \"kubernetes.io/projected/db252401-bd9d-4cdc-b400-3e07d2ae2377-kube-api-access-5nbqb\") pod \"db252401-bd9d-4cdc-b400-3e07d2ae2377\" (UID: \"db252401-bd9d-4cdc-b400-3e07d2ae2377\") "
Sep 05 02:29:31.804424 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.804397    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db252401-bd9d-4cdc-b400-3e07d2ae2377-cert\") pod \"db252401-bd9d-4cdc-b400-3e07d2ae2377\" (UID: \"db252401-bd9d-4cdc-b400-3e07d2ae2377\") "
Sep 05 02:29:31.806466 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.806418    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db252401-bd9d-4cdc-b400-3e07d2ae2377-cert" (OuterVolumeSpecName: "cert") pod "db252401-bd9d-4cdc-b400-3e07d2ae2377" (UID: "db252401-bd9d-4cdc-b400-3e07d2ae2377"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:29:31.806569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.806459    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db252401-bd9d-4cdc-b400-3e07d2ae2377-kube-api-access-5nbqb" (OuterVolumeSpecName: "kube-api-access-5nbqb") pod "db252401-bd9d-4cdc-b400-3e07d2ae2377" (UID: "db252401-bd9d-4cdc-b400-3e07d2ae2377"). InnerVolumeSpecName "kube-api-access-5nbqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:29:31.897872 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.897844    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-tjpj4"
Sep 05 02:29:31.905744 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.905721    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nbqb\" (UniqueName: \"kubernetes.io/projected/db252401-bd9d-4cdc-b400-3e07d2ae2377-kube-api-access-5nbqb\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:29:31.905744 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.905744    2557 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db252401-bd9d-4cdc-b400-3e07d2ae2377-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:29:31.960282 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.960254    2557 generic.go:356] "Generic (PLEG): container finished" podID="db252401-bd9d-4cdc-b400-3e07d2ae2377" containerID="0ed04e50ec43d92102bf99d477084c6393b2bb17eb62a1bff148609e49e85116" exitCode=0
Sep 05 02:29:31.960428 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.960315    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn"
Sep 05 02:29:31.960428 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.960321    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn" event={"ID":"db252401-bd9d-4cdc-b400-3e07d2ae2377","Type":"ContainerDied","Data":"0ed04e50ec43d92102bf99d477084c6393b2bb17eb62a1bff148609e49e85116"}
Sep 05 02:29:31.960428 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.960362    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-khtsn" event={"ID":"db252401-bd9d-4cdc-b400-3e07d2ae2377","Type":"ContainerDied","Data":"4d89f3cb75d543b2d4c25693aaf18840dd011ac7ebd834a822dea7d9b5cd9b5b"}
Sep 05 02:29:31.960428 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.960383    2557 scope.go:117] "RemoveContainer" containerID="0ed04e50ec43d92102bf99d477084c6393b2bb17eb62a1bff148609e49e85116"
Sep 05 02:29:31.970708 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.970688    2557 scope.go:117] "RemoveContainer" containerID="0ed04e50ec43d92102bf99d477084c6393b2bb17eb62a1bff148609e49e85116"
Sep 05 02:29:31.970999 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:29:31.970977    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed04e50ec43d92102bf99d477084c6393b2bb17eb62a1bff148609e49e85116\": container with ID starting with 0ed04e50ec43d92102bf99d477084c6393b2bb17eb62a1bff148609e49e85116 not found: ID does not exist" containerID="0ed04e50ec43d92102bf99d477084c6393b2bb17eb62a1bff148609e49e85116"
Sep 05 02:29:31.971037 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.971011    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed04e50ec43d92102bf99d477084c6393b2bb17eb62a1bff148609e49e85116"} err="failed to get container status \"0ed04e50ec43d92102bf99d477084c6393b2bb17eb62a1bff148609e49e85116\": rpc error: code = NotFound desc = could not find container \"0ed04e50ec43d92102bf99d477084c6393b2bb17eb62a1bff148609e49e85116\": container with ID starting with 0ed04e50ec43d92102bf99d477084c6393b2bb17eb62a1bff148609e49e85116 not found: ID does not exist"
Sep 05 02:29:31.986117 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.986095    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-khtsn"]
Sep 05 02:29:31.989581 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:31.989561    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-khtsn"]
Sep 05 02:29:32.166678 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:32.166651    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-577b5fb6f8-tjpj4"]
Sep 05 02:29:32.168880 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:29:32.168852    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a6af99_2320_483c_af78_37c506467fc4.slice/crio-bcc6550608d4550b78f29f31775306691dbbb880db888107222957947faac291 WatchSource:0}: Error finding container bcc6550608d4550b78f29f31775306691dbbb880db888107222957947faac291: Status 404 returned error can't find the container with id bcc6550608d4550b78f29f31775306691dbbb880db888107222957947faac291
Sep 05 02:29:32.170180 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:32.170160    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:29:32.966117 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:32.966078    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-tjpj4" event={"ID":"44a6af99-2320-483c-af78-37c506467fc4","Type":"ContainerStarted","Data":"bcc6550608d4550b78f29f31775306691dbbb880db888107222957947faac291"}
Sep 05 02:29:32.991888 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:32.991859    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db252401-bd9d-4cdc-b400-3e07d2ae2377" path="/var/lib/kubelet/pods/db252401-bd9d-4cdc-b400-3e07d2ae2377/volumes"
Sep 05 02:29:33.973188 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:33.973154    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-577b5fb6f8-tjpj4" event={"ID":"44a6af99-2320-483c-af78-37c506467fc4","Type":"ContainerStarted","Data":"a55868a2edca83dbe760a5223fb19d635916fad5e6aac27e27a62c781f4b9e2a"}
Sep 05 02:29:33.973576 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:33.973274    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-577b5fb6f8-tjpj4"
Sep 05 02:29:33.991534 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:29:33.991489    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-577b5fb6f8-tjpj4" podStartSLOduration=2.174280431 podStartE2EDuration="2.991475269s" totalImagesPullingTime="817.194838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:29:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:29:33.989144491 +0000 UTC m=+677.544300799" watchObservedRunningTime="2026-09-05 02:29:33.991475269 +0000 UTC m=+677.546631577"
Sep 05 02:30:04.983391 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:04.983358    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-577b5fb6f8-tjpj4"
Sep 05 02:30:06.297914 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.297881    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:30:06.343905 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.343860    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-l874f"]
Sep 05 02:30:06.344634 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.344609    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db252401-bd9d-4cdc-b400-3e07d2ae2377" containerName="manager"
Sep 05 02:30:06.344634 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.344634    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="db252401-bd9d-4cdc-b400-3e07d2ae2377" containerName="manager"
Sep 05 02:30:06.344829 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.344814    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="db252401-bd9d-4cdc-b400-3e07d2ae2377" containerName="manager"
Sep 05 02:30:06.348991 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.348971    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-l874f"
Sep 05 02:30:06.351880 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.351856    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 05 02:30:06.352003 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.351879    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-85dtv\""
Sep 05 02:30:06.352548 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.352530    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-cd6lx"]
Sep 05 02:30:06.355837 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.355807    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:06.357161 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.357141    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-l874f"]
Sep 05 02:30:06.357754 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.357732    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 05 02:30:06.357856 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.357794    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 05 02:30:06.357856 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.357828    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-bgfmk\""
Sep 05 02:30:06.373583 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.373549    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-cd6lx"]
Sep 05 02:30:06.503724 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.503693    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b-tls-certs\") pod \"model-serving-api-7b954ffc95-l874f\" (UID: \"ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b\") " pod="kserve/model-serving-api-7b954ffc95-l874f"
Sep 05 02:30:06.503895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.503729    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5233c7a9-331e-49ef-9cc9-5c99037d4c49-metrics-certs\") pod \"odh-model-controller-7b966d7984-cd6lx\" (UID: \"5233c7a9-331e-49ef-9cc9-5c99037d4c49\") " pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:06.503895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.503757    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f2g6\" (UniqueName: \"kubernetes.io/projected/ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b-kube-api-access-5f2g6\") pod \"model-serving-api-7b954ffc95-l874f\" (UID: \"ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b\") " pod="kserve/model-serving-api-7b954ffc95-l874f"
Sep 05 02:30:06.503895 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.503851    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5233c7a9-331e-49ef-9cc9-5c99037d4c49-cert\") pod \"odh-model-controller-7b966d7984-cd6lx\" (UID: \"5233c7a9-331e-49ef-9cc9-5c99037d4c49\") " pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:06.504024 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.503900    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x99r5\" (UniqueName: \"kubernetes.io/projected/5233c7a9-331e-49ef-9cc9-5c99037d4c49-kube-api-access-x99r5\") pod \"odh-model-controller-7b966d7984-cd6lx\" (UID: \"5233c7a9-331e-49ef-9cc9-5c99037d4c49\") " pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:06.605036 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.604993    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x99r5\" (UniqueName: \"kubernetes.io/projected/5233c7a9-331e-49ef-9cc9-5c99037d4c49-kube-api-access-x99r5\") pod \"odh-model-controller-7b966d7984-cd6lx\" (UID: \"5233c7a9-331e-49ef-9cc9-5c99037d4c49\") " pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:06.605205 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.605086    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b-tls-certs\") pod \"model-serving-api-7b954ffc95-l874f\" (UID: \"ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b\") " pod="kserve/model-serving-api-7b954ffc95-l874f"
Sep 05 02:30:06.605205 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.605114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5233c7a9-331e-49ef-9cc9-5c99037d4c49-metrics-certs\") pod \"odh-model-controller-7b966d7984-cd6lx\" (UID: \"5233c7a9-331e-49ef-9cc9-5c99037d4c49\") " pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:06.605205 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.605155    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f2g6\" (UniqueName: \"kubernetes.io/projected/ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b-kube-api-access-5f2g6\") pod \"model-serving-api-7b954ffc95-l874f\" (UID: \"ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b\") " pod="kserve/model-serving-api-7b954ffc95-l874f"
Sep 05 02:30:06.605205 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.605203    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5233c7a9-331e-49ef-9cc9-5c99037d4c49-cert\") pod \"odh-model-controller-7b966d7984-cd6lx\" (UID: \"5233c7a9-331e-49ef-9cc9-5c99037d4c49\") " pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:06.605424 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:30:06.605314    2557 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 05 02:30:06.605424 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:30:06.605376    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5233c7a9-331e-49ef-9cc9-5c99037d4c49-cert podName:5233c7a9-331e-49ef-9cc9-5c99037d4c49 nodeName:}" failed. No retries permitted until 2026-09-05 02:30:07.1053565 +0000 UTC m=+710.660512792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5233c7a9-331e-49ef-9cc9-5c99037d4c49-cert") pod "odh-model-controller-7b966d7984-cd6lx" (UID: "5233c7a9-331e-49ef-9cc9-5c99037d4c49") : secret "odh-model-controller-webhook-cert" not found
Sep 05 02:30:06.607619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.607595    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5233c7a9-331e-49ef-9cc9-5c99037d4c49-metrics-certs\") pod \"odh-model-controller-7b966d7984-cd6lx\" (UID: \"5233c7a9-331e-49ef-9cc9-5c99037d4c49\") " pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:06.607744 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.607665    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b-tls-certs\") pod \"model-serving-api-7b954ffc95-l874f\" (UID: \"ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b\") " pod="kserve/model-serving-api-7b954ffc95-l874f"
Sep 05 02:30:06.615572 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.615550    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x99r5\" (UniqueName: \"kubernetes.io/projected/5233c7a9-331e-49ef-9cc9-5c99037d4c49-kube-api-access-x99r5\") pod \"odh-model-controller-7b966d7984-cd6lx\" (UID: \"5233c7a9-331e-49ef-9cc9-5c99037d4c49\") " pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:06.615889 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.615867    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f2g6\" (UniqueName: \"kubernetes.io/projected/ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b-kube-api-access-5f2g6\") pod \"model-serving-api-7b954ffc95-l874f\" (UID: \"ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b\") " pod="kserve/model-serving-api-7b954ffc95-l874f"
Sep 05 02:30:06.661003 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.660980    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-l874f"
Sep 05 02:30:06.948016 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:06.947978    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-l874f"]
Sep 05 02:30:06.950114 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:30:06.950083    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5fbaa7_2a65_49e8_91c6_23ad1d73e63b.slice/crio-aeb70e4f900b593526f015f3e6f79225e2bd24d810fafe677e52d45ba7e796f6 WatchSource:0}: Error finding container aeb70e4f900b593526f015f3e6f79225e2bd24d810fafe677e52d45ba7e796f6: Status 404 returned error can't find the container with id aeb70e4f900b593526f015f3e6f79225e2bd24d810fafe677e52d45ba7e796f6
Sep 05 02:30:07.110649 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:07.110613    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5233c7a9-331e-49ef-9cc9-5c99037d4c49-cert\") pod \"odh-model-controller-7b966d7984-cd6lx\" (UID: \"5233c7a9-331e-49ef-9cc9-5c99037d4c49\") " pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:07.110826 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:30:07.110765    2557 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 05 02:30:07.110896 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:30:07.110831    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5233c7a9-331e-49ef-9cc9-5c99037d4c49-cert podName:5233c7a9-331e-49ef-9cc9-5c99037d4c49 nodeName:}" failed. No retries permitted until 2026-09-05 02:30:08.110810234 +0000 UTC m=+711.665966533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5233c7a9-331e-49ef-9cc9-5c99037d4c49-cert") pod "odh-model-controller-7b966d7984-cd6lx" (UID: "5233c7a9-331e-49ef-9cc9-5c99037d4c49") : secret "odh-model-controller-webhook-cert" not found
Sep 05 02:30:07.128688 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:07.128663    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-l874f" event={"ID":"ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b","Type":"ContainerStarted","Data":"aeb70e4f900b593526f015f3e6f79225e2bd24d810fafe677e52d45ba7e796f6"}
Sep 05 02:30:08.118095 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:08.118062    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5233c7a9-331e-49ef-9cc9-5c99037d4c49-cert\") pod \"odh-model-controller-7b966d7984-cd6lx\" (UID: \"5233c7a9-331e-49ef-9cc9-5c99037d4c49\") " pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:08.120467 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:08.120421    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5233c7a9-331e-49ef-9cc9-5c99037d4c49-cert\") pod \"odh-model-controller-7b966d7984-cd6lx\" (UID: \"5233c7a9-331e-49ef-9cc9-5c99037d4c49\") " pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:08.167718 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:08.167689    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:08.214416 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:08.214369    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:30:08.447657 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:08.447554    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-cd6lx"]
Sep 05 02:30:08.570205 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:30:08.570173    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5233c7a9_331e_49ef_9cc9_5c99037d4c49.slice/crio-8c9dd3761febbd2b6371066db8418c2627258d67155e7f8280995a431b543540 WatchSource:0}: Error finding container 8c9dd3761febbd2b6371066db8418c2627258d67155e7f8280995a431b543540: Status 404 returned error can't find the container with id 8c9dd3761febbd2b6371066db8418c2627258d67155e7f8280995a431b543540
Sep 05 02:30:09.142564 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:09.142483    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-cd6lx" event={"ID":"5233c7a9-331e-49ef-9cc9-5c99037d4c49","Type":"ContainerStarted","Data":"8c9dd3761febbd2b6371066db8418c2627258d67155e7f8280995a431b543540"}
Sep 05 02:30:09.143923 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:09.143899    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-l874f" event={"ID":"ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b","Type":"ContainerStarted","Data":"0ccb37fb809873e3f3e5cd22a7638898e9fb37c742366052ea69c74f3c353c47"}
Sep 05 02:30:09.144012 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:09.143951    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-l874f"
Sep 05 02:30:09.162660 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:09.162619    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-l874f" podStartSLOduration=1.496285648 podStartE2EDuration="3.162604648s" totalImagesPullingTime="1.666319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:30:09.16135245 +0000 UTC m=+712.716508761" watchObservedRunningTime="2026-09-05 02:30:09.162604648 +0000 UTC m=+712.717761013"
Sep 05 02:30:11.156868 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:11.156834    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-cd6lx" event={"ID":"5233c7a9-331e-49ef-9cc9-5c99037d4c49","Type":"ContainerStarted","Data":"44ddf8f2d4ec315257e75782fad63ec4238d1c467b95f9a302c319884ac6e2bb"}
Sep 05 02:30:11.157171 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:11.156938    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:11.175250 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:11.175206    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-cd6lx" podStartSLOduration=2.677766519 podStartE2EDuration="5.175191665s" totalImagesPullingTime="2.497425146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:30:11.172410522 +0000 UTC m=+714.727566831" watchObservedRunningTime="2026-09-05 02:30:11.175191665 +0000 UTC m=+714.730347973"
Sep 05 02:30:20.154030 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:20.154002    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-l874f"
Sep 05 02:30:22.164648 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:22.164611    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-cd6lx"
Sep 05 02:30:23.007249 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:23.007217    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-pggpg"]
Sep 05 02:30:23.011650 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:23.011628    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pggpg"
Sep 05 02:30:23.013641 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:23.013622    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 05 02:30:23.017998 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:23.017977    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pggpg"]
Sep 05 02:30:23.149631 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:23.149596    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc86525-561c-4d9f-87f9-0df00516dc33-model-cache\") pod \"s3-init-pggpg\" (UID: \"dcc86525-561c-4d9f-87f9-0df00516dc33\") " pod="kserve/s3-init-pggpg"
Sep 05 02:30:23.149631 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:23.149633    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lml2j\" (UniqueName: \"kubernetes.io/projected/dcc86525-561c-4d9f-87f9-0df00516dc33-kube-api-access-lml2j\") pod \"s3-init-pggpg\" (UID: \"dcc86525-561c-4d9f-87f9-0df00516dc33\") " pod="kserve/s3-init-pggpg"
Sep 05 02:30:23.250356 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:23.250327    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc86525-561c-4d9f-87f9-0df00516dc33-model-cache\") pod \"s3-init-pggpg\" (UID: \"dcc86525-561c-4d9f-87f9-0df00516dc33\") " pod="kserve/s3-init-pggpg"
Sep 05 02:30:23.250356 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:23.250359    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lml2j\" (UniqueName: \"kubernetes.io/projected/dcc86525-561c-4d9f-87f9-0df00516dc33-kube-api-access-lml2j\") pod \"s3-init-pggpg\" (UID: \"dcc86525-561c-4d9f-87f9-0df00516dc33\") " pod="kserve/s3-init-pggpg"
Sep 05 02:30:23.250735 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:23.250696    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc86525-561c-4d9f-87f9-0df00516dc33-model-cache\") pod \"s3-init-pggpg\" (UID: \"dcc86525-561c-4d9f-87f9-0df00516dc33\") " pod="kserve/s3-init-pggpg"
Sep 05 02:30:23.258211 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:23.258152    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lml2j\" (UniqueName: \"kubernetes.io/projected/dcc86525-561c-4d9f-87f9-0df00516dc33-kube-api-access-lml2j\") pod \"s3-init-pggpg\" (UID: \"dcc86525-561c-4d9f-87f9-0df00516dc33\") " pod="kserve/s3-init-pggpg"
Sep 05 02:30:23.321852 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:23.321817    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pggpg"
Sep 05 02:30:23.590638 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:23.590610    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pggpg"]
Sep 05 02:30:23.591948 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:30:23.591918    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcc86525_561c_4d9f_87f9_0df00516dc33.slice/crio-a3ef9926a9806dac1b4fd8834c0c04e6f45a360667d320512e2a5a70251b890c WatchSource:0}: Error finding container a3ef9926a9806dac1b4fd8834c0c04e6f45a360667d320512e2a5a70251b890c: Status 404 returned error can't find the container with id a3ef9926a9806dac1b4fd8834c0c04e6f45a360667d320512e2a5a70251b890c
Sep 05 02:30:24.224049 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:24.224005    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pggpg" event={"ID":"dcc86525-561c-4d9f-87f9-0df00516dc33","Type":"ContainerStarted","Data":"a3ef9926a9806dac1b4fd8834c0c04e6f45a360667d320512e2a5a70251b890c"}
Sep 05 02:30:27.243944 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:27.243911    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pggpg" event={"ID":"dcc86525-561c-4d9f-87f9-0df00516dc33","Type":"ContainerStarted","Data":"d5b5667d0150e584ad76fbe18b6637f274a5c4e638f9653ca35e292c1c0736fe"}
Sep 05 02:30:29.256164 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:29.256073    2557 generic.go:356] "Generic (PLEG): container finished" podID="dcc86525-561c-4d9f-87f9-0df00516dc33" containerID="d5b5667d0150e584ad76fbe18b6637f274a5c4e638f9653ca35e292c1c0736fe" exitCode=0
Sep 05 02:30:29.256164 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:29.256152    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pggpg" event={"ID":"dcc86525-561c-4d9f-87f9-0df00516dc33","Type":"ContainerDied","Data":"d5b5667d0150e584ad76fbe18b6637f274a5c4e638f9653ca35e292c1c0736fe"}
Sep 05 02:30:34.294209 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:34.294173    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pggpg" event={"ID":"dcc86525-561c-4d9f-87f9-0df00516dc33","Type":"ContainerStarted","Data":"1e0eae8ead19d2d83a1e635f4764e4a40cba8601bb286ad7f82a919857c4526c"}
Sep 05 02:30:34.312348 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:30:34.312298    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-pggpg" podStartSLOduration=2.782353343 podStartE2EDuration="12.312282866s" totalImagesPullingTime="7.50887834s" totalInitContainerRuntime="2.021051183s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:30:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:30:34.31028366 +0000 UTC m=+737.865439967" watchObservedRunningTime="2026-09-05 02:30:34.312282866 +0000 UTC m=+737.867439173"
Sep 05 02:31:16.832554 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:16.832520    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:32.877012 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:32.876966    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:38.630500 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:38.630465    2557 generic.go:356] "Generic (PLEG): container finished" podID="dcc86525-561c-4d9f-87f9-0df00516dc33" containerID="1e0eae8ead19d2d83a1e635f4764e4a40cba8601bb286ad7f82a919857c4526c" exitCode=0
Sep 05 02:31:38.630907 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:38.630531    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pggpg" event={"ID":"dcc86525-561c-4d9f-87f9-0df00516dc33","Type":"ContainerDied","Data":"1e0eae8ead19d2d83a1e635f4764e4a40cba8601bb286ad7f82a919857c4526c"}
Sep 05 02:31:39.761638 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:39.761613    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pggpg"
Sep 05 02:31:39.935304 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:39.935215    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lml2j\" (UniqueName: \"kubernetes.io/projected/dcc86525-561c-4d9f-87f9-0df00516dc33-kube-api-access-lml2j\") pod \"dcc86525-561c-4d9f-87f9-0df00516dc33\" (UID: \"dcc86525-561c-4d9f-87f9-0df00516dc33\") "
Sep 05 02:31:39.935304 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:39.935278    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc86525-561c-4d9f-87f9-0df00516dc33-model-cache\") pod \"dcc86525-561c-4d9f-87f9-0df00516dc33\" (UID: \"dcc86525-561c-4d9f-87f9-0df00516dc33\") "
Sep 05 02:31:39.937328 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:39.937296    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc86525-561c-4d9f-87f9-0df00516dc33-kube-api-access-lml2j" (OuterVolumeSpecName: "kube-api-access-lml2j") pod "dcc86525-561c-4d9f-87f9-0df00516dc33" (UID: "dcc86525-561c-4d9f-87f9-0df00516dc33"). InnerVolumeSpecName "kube-api-access-lml2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:31:39.953818 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:39.953784    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc86525-561c-4d9f-87f9-0df00516dc33-model-cache" (OuterVolumeSpecName: "model-cache") pod "dcc86525-561c-4d9f-87f9-0df00516dc33" (UID: "dcc86525-561c-4d9f-87f9-0df00516dc33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:31:40.036562 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:40.036527    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lml2j\" (UniqueName: \"kubernetes.io/projected/dcc86525-561c-4d9f-87f9-0df00516dc33-kube-api-access-lml2j\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:31:40.036562 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:40.036560    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc86525-561c-4d9f-87f9-0df00516dc33-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:31:40.642402 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:40.642374    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pggpg"
Sep 05 02:31:40.642609 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:40.642367    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pggpg" event={"ID":"dcc86525-561c-4d9f-87f9-0df00516dc33","Type":"ContainerDied","Data":"a3ef9926a9806dac1b4fd8834c0c04e6f45a360667d320512e2a5a70251b890c"}
Sep 05 02:31:40.642609 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:40.642503    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3ef9926a9806dac1b4fd8834c0c04e6f45a360667d320512e2a5a70251b890c"
Sep 05 02:31:41.027606 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:41.027528    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:41.125308 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:41.125272    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:45.075597 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:45.075547    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:53.422043 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:53.422003    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:53.517705 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:53.517677    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:58.397169 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.397126    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"]
Sep 05 02:31:58.397651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.397507    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcc86525-561c-4d9f-87f9-0df00516dc33" containerName="download-hf-model"
Sep 05 02:31:58.397651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.397520    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcc86525-561c-4d9f-87f9-0df00516dc33" containerName="download-hf-model"
Sep 05 02:31:58.397651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.397528    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcc86525-561c-4d9f-87f9-0df00516dc33" containerName="s3-init"
Sep 05 02:31:58.397651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.397534    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcc86525-561c-4d9f-87f9-0df00516dc33" containerName="s3-init"
Sep 05 02:31:58.397651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.397587    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcc86525-561c-4d9f-87f9-0df00516dc33" containerName="s3-init"
Sep 05 02:31:58.401256 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.401221    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.403649 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.403613    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:31:58.403793 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.403668    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 05 02:31:58.403793 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.403743    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 05 02:31:58.403793 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.403746    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-nqqdf\""
Sep 05 02:31:58.404002 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.403882    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 05 02:31:58.404058 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.404027    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 05 02:31:58.414533 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.414508    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"]
Sep 05 02:31:58.429606 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.429572    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:31:58.490813 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.490774    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.490813 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.490814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.491021 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.490947    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.491021 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.490989    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/279be1f8-14e4-43af-b769-30b1106a5475-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.491021 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.491012    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.491139 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.491048    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.491139 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.491066    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279be1f8-14e4-43af-b769-30b1106a5475-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.491139 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.491105    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtf7j\" (UniqueName: \"kubernetes.io/projected/279be1f8-14e4-43af-b769-30b1106a5475-kube-api-access-rtf7j\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.592481 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.592417    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.592481 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.592481    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279be1f8-14e4-43af-b769-30b1106a5475-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.592814 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.592508    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtf7j\" (UniqueName: \"kubernetes.io/projected/279be1f8-14e4-43af-b769-30b1106a5475-kube-api-access-rtf7j\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.592814 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.592530    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.592814 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.592547    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.592814 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.592616    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.592814 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.592647    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/279be1f8-14e4-43af-b769-30b1106a5475-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.592814 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.592679    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.593211 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.593183    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.593300 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.593222    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.593300 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.593263    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.593393 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.593300    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.593393 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.593384    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/279be1f8-14e4-43af-b769-30b1106a5475-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.594888 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.594866    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.595181 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.595159    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279be1f8-14e4-43af-b769-30b1106a5475-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.602491 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.602470    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtf7j\" (UniqueName: \"kubernetes.io/projected/279be1f8-14e4-43af-b769-30b1106a5475-kube-api-access-rtf7j\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:58.712011 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:58.711890    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:31:59.006702 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:59.006676    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"]
Sep 05 02:31:59.008638 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:31:59.008609    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279be1f8_14e4_43af_b769_30b1106a5475.slice/crio-a3ac7564ebdf603dfb2b302d476dbd673e32346258b49f269114bf4e470de17b WatchSource:0}: Error finding container a3ac7564ebdf603dfb2b302d476dbd673e32346258b49f269114bf4e470de17b: Status 404 returned error can't find the container with id a3ac7564ebdf603dfb2b302d476dbd673e32346258b49f269114bf4e470de17b
Sep 05 02:31:59.749160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:59.748289    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" event={"ID":"279be1f8-14e4-43af-b769-30b1106a5475","Type":"ContainerStarted","Data":"4ba9cf640c54fc1aa76c8d73f163f570e2bff291b283e0eb814cc577ba45c961"}
Sep 05 02:31:59.749160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:31:59.748356    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" event={"ID":"279be1f8-14e4-43af-b769-30b1106a5475","Type":"ContainerStarted","Data":"a3ac7564ebdf603dfb2b302d476dbd673e32346258b49f269114bf4e470de17b"}
Sep 05 02:32:01.765002 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:01.764964    2557 generic.go:356] "Generic (PLEG): container finished" podID="279be1f8-14e4-43af-b769-30b1106a5475" containerID="4ba9cf640c54fc1aa76c8d73f163f570e2bff291b283e0eb814cc577ba45c961" exitCode=0
Sep 05 02:32:01.765492 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:01.765128    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" event={"ID":"279be1f8-14e4-43af-b769-30b1106a5475","Type":"ContainerDied","Data":"4ba9cf640c54fc1aa76c8d73f163f570e2bff291b283e0eb814cc577ba45c961"}
Sep 05 02:32:07.317632 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:07.317594    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:08.083785 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:08.083605    2557 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 05 02:32:08.083785 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:08.083679    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm" podUID="82df5585-ce83-449d-8931-c22fd607d1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 05 02:32:09.846464 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:09.846024    2557 generic.go:356] "Generic (PLEG): container finished" podID="82df5585-ce83-449d-8931-c22fd607d1f4" containerID="2f56ec957f9e0ef6f5f2467f41f36b343ed5d62cba84f9f40b5d918d4d4f4599" exitCode=0
Sep 05 02:32:09.846464 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:09.846175    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm" event={"ID":"82df5585-ce83-449d-8931-c22fd607d1f4","Type":"ContainerDied","Data":"2f56ec957f9e0ef6f5f2467f41f36b343ed5d62cba84f9f40b5d918d4d4f4599"}
Sep 05 02:32:09.847268 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:09.847025    2557 scope.go:117] "RemoveContainer" containerID="2f56ec957f9e0ef6f5f2467f41f36b343ed5d62cba84f9f40b5d918d4d4f4599"
Sep 05 02:32:09.849209 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:09.848842    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 02:32:09.849209 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:09.848936    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 02:32:09.849209 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:09.848984    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 02:32:10.872884 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:10.872588    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm" event={"ID":"82df5585-ce83-449d-8931-c22fd607d1f4","Type":"ContainerStarted","Data":"9cc6ac02650eaec19c29125a9850df483eb4689da5758cd74903e39b93f3667c"}
Sep 05 02:32:11.814563 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:11.814322    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:32:11.814759 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:11.814597    2557 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 05 02:32:11.814759 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:11.814653    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm" podUID="82df5585-ce83-449d-8931-c22fd607d1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 05 02:32:11.903465 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:11.903341    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:12.135617 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.133763    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"]
Sep 05 02:32:12.159459 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.147026    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"
Sep 05 02:32:12.165294 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.162884    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 05 02:32:12.165294 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.163195    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 05 02:32:12.165294 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.163409    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 05 02:32:12.165294 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.164186    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-2prbc\""
Sep 05 02:32:12.165294 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.164426    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-vdvdj\""
Sep 05 02:32:12.167271 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.167227    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"]
Sep 05 02:32:12.181852 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.181622    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:12.283011 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.282914    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcj5z\" (UniqueName: \"kubernetes.io/projected/783a0c6c-c678-429d-a1b2-7aa987eb9248-kube-api-access-dcj5z\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc\" (UID: \"783a0c6c-c678-429d-a1b2-7aa987eb9248\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"
Sep 05 02:32:12.283011 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.282981    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783a0c6c-c678-429d-a1b2-7aa987eb9248-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc\" (UID: \"783a0c6c-c678-429d-a1b2-7aa987eb9248\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"
Sep 05 02:32:12.384126 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.384071    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcj5z\" (UniqueName: \"kubernetes.io/projected/783a0c6c-c678-429d-a1b2-7aa987eb9248-kube-api-access-dcj5z\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc\" (UID: \"783a0c6c-c678-429d-a1b2-7aa987eb9248\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"
Sep 05 02:32:12.384319 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.384140    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783a0c6c-c678-429d-a1b2-7aa987eb9248-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc\" (UID: \"783a0c6c-c678-429d-a1b2-7aa987eb9248\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"
Sep 05 02:32:12.389230 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.389128    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783a0c6c-c678-429d-a1b2-7aa987eb9248-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc\" (UID: \"783a0c6c-c678-429d-a1b2-7aa987eb9248\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"
Sep 05 02:32:12.395127 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.395066    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcj5z\" (UniqueName: \"kubernetes.io/projected/783a0c6c-c678-429d-a1b2-7aa987eb9248-kube-api-access-dcj5z\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc\" (UID: \"783a0c6c-c678-429d-a1b2-7aa987eb9248\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"
Sep 05 02:32:12.490923 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.490859    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"
Sep 05 02:32:12.814384 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.814294    2557 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 05 02:32:12.814384 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:12.814367    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm" podUID="82df5585-ce83-449d-8931-c22fd607d1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 05 02:32:13.814141 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:13.814100    2557 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 05 02:32:13.814634 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:13.814180    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm" podUID="82df5585-ce83-449d-8931-c22fd607d1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 05 02:32:14.814243 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:14.814208    2557 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 05 02:32:14.814774 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:14.814266    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm" podUID="82df5585-ce83-449d-8931-c22fd607d1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 05 02:32:15.813666 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:15.813622    2557 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 05 02:32:15.813838 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:15.813694    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm" podUID="82df5585-ce83-449d-8931-c22fd607d1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 05 02:32:17.655114 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:17.655067    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:32:17.656134 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:17.655848    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:32:17.657114 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:17.657091    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm"
Sep 05 02:32:19.396309 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:19.396279    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"]
Sep 05 02:32:19.397912 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:32:19.397883    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783a0c6c_c678_429d_a1b2_7aa987eb9248.slice/crio-186f8aca8c62b6dfc6215c3dec2c4b889d4715b2a0669bb02cc06c942f448361 WatchSource:0}: Error finding container 186f8aca8c62b6dfc6215c3dec2c4b889d4715b2a0669bb02cc06c942f448361: Status 404 returned error can't find the container with id 186f8aca8c62b6dfc6215c3dec2c4b889d4715b2a0669bb02cc06c942f448361
Sep 05 02:32:19.937391 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:19.937355    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc" event={"ID":"783a0c6c-c678-429d-a1b2-7aa987eb9248","Type":"ContainerStarted","Data":"186f8aca8c62b6dfc6215c3dec2c4b889d4715b2a0669bb02cc06c942f448361"}
Sep 05 02:32:29.176298 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:29.176249    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:29.634184 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:29.634146    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:29.734728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:29.734689    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:35.293400 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:35.293273    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:32:36.039967 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:36.038951    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc" event={"ID":"783a0c6c-c678-429d-a1b2-7aa987eb9248","Type":"ContainerStarted","Data":"ca3d4a057acac34e423b60137d2653f27fc26e012775e11c660c909f36282ea1"}
Sep 05 02:32:36.039967 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:36.039838    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"
Sep 05 02:32:36.059649 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:36.059594    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc" podStartSLOduration=7.704853573 podStartE2EDuration="24.059579742s" totalImagesPullingTime="16.354726169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:32:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:32:36.056711716 +0000 UTC m=+859.611868026" watchObservedRunningTime="2026-09-05 02:32:36.059579742 +0000 UTC m=+859.614736051"
Sep 05 02:32:37.048746 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:37.048709    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" event={"ID":"279be1f8-14e4-43af-b769-30b1106a5475","Type":"ContainerStarted","Data":"6bf034201807fbbc6598869d2f15d6d8dbc8b612d76eedb0967413d68ac84e3c"}
Sep 05 02:32:37.071682 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:37.071623    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" podStartSLOduration=2.168802157 podStartE2EDuration="39.071606915s" totalImagesPullingTime="34.352264879s" totalInitContainerRuntime="2.550539879s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:32:37.068929601 +0000 UTC m=+860.624085909" watchObservedRunningTime="2026-09-05 02:32:37.071606915 +0000 UTC m=+860.626763272"
Sep 05 02:32:37.712698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:37.712662    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:32:38.712037 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:38.711996    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:32:38.713505 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:38.713474    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 05 02:32:48.713225 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:48.713176    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 05 02:32:58.712433 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:32:58.712385    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 05 02:33:08.055385 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:08.055349    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"
Sep 05 02:33:08.712377 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:08.712329    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 05 02:33:14.943383 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:14.943347    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"]
Sep 05 02:33:14.946648 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:14.946621    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:15.014778 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:15.014741    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:16.904302 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:16.904270    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:33:16.907427 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:16.907405    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:33:18.055401 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:18.055361    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc" podUID="783a0c6c-c678-429d-a1b2-7aa987eb9248" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:33:18.712310 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:18.712260    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 05 02:33:28.055927 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:28.055886    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc" podUID="783a0c6c-c678-429d-a1b2-7aa987eb9248" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:33:28.712354 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:28.712299    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 05 02:33:29.973578 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:29.973529    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc" podUID="783a0c6c-c678-429d-a1b2-7aa987eb9248" containerName="main" containerID="cri-o://ca3d4a057acac34e423b60137d2653f27fc26e012775e11c660c909f36282ea1" gracePeriod=45
Sep 05 02:33:38.055788 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:38.055749    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc" podUID="783a0c6c-c678-429d-a1b2-7aa987eb9248" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:33:38.056249 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:38.055850    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"
Sep 05 02:33:38.713327 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:38.713281    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 05 02:33:38.830227 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:38.830184    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:33:48.055623 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:48.055576    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc" podUID="783a0c6c-c678-429d-a1b2-7aa987eb9248" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:33:48.712582 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:48.712540    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 05 02:33:58.062679 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:58.062633    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc" podUID="783a0c6c-c678-429d-a1b2-7aa987eb9248" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:33:58.713363 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:33:58.713315    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 05 02:34:00.234776 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.234718    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"
Sep 05 02:34:00.287010 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.286973    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcj5z\" (UniqueName: \"kubernetes.io/projected/783a0c6c-c678-429d-a1b2-7aa987eb9248-kube-api-access-dcj5z\") pod \"783a0c6c-c678-429d-a1b2-7aa987eb9248\" (UID: \"783a0c6c-c678-429d-a1b2-7aa987eb9248\") "
Sep 05 02:34:00.287204 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.287043    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783a0c6c-c678-429d-a1b2-7aa987eb9248-tls-certs\") pod \"783a0c6c-c678-429d-a1b2-7aa987eb9248\" (UID: \"783a0c6c-c678-429d-a1b2-7aa987eb9248\") "
Sep 05 02:34:00.289266 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.289230    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783a0c6c-c678-429d-a1b2-7aa987eb9248-kube-api-access-dcj5z" (OuterVolumeSpecName: "kube-api-access-dcj5z") pod "783a0c6c-c678-429d-a1b2-7aa987eb9248" (UID: "783a0c6c-c678-429d-a1b2-7aa987eb9248"). InnerVolumeSpecName "kube-api-access-dcj5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:34:00.289266 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.289239    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/783a0c6c-c678-429d-a1b2-7aa987eb9248-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "783a0c6c-c678-429d-a1b2-7aa987eb9248" (UID: "783a0c6c-c678-429d-a1b2-7aa987eb9248"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:34:00.388166 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.388132    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783a0c6c-c678-429d-a1b2-7aa987eb9248-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:34:00.388166 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.388159    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcj5z\" (UniqueName: \"kubernetes.io/projected/783a0c6c-c678-429d-a1b2-7aa987eb9248-kube-api-access-dcj5z\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:34:00.547890 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.547798    2557 generic.go:356] "Generic (PLEG): container finished" podID="783a0c6c-c678-429d-a1b2-7aa987eb9248" containerID="ca3d4a057acac34e423b60137d2653f27fc26e012775e11c660c909f36282ea1" exitCode=0
Sep 05 02:34:00.547890 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.547839    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"
Sep 05 02:34:00.547890 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.547878    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc" event={"ID":"783a0c6c-c678-429d-a1b2-7aa987eb9248","Type":"ContainerDied","Data":"ca3d4a057acac34e423b60137d2653f27fc26e012775e11c660c909f36282ea1"}
Sep 05 02:34:00.548235 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.547936    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc" event={"ID":"783a0c6c-c678-429d-a1b2-7aa987eb9248","Type":"ContainerDied","Data":"186f8aca8c62b6dfc6215c3dec2c4b889d4715b2a0669bb02cc06c942f448361"}
Sep 05 02:34:00.548235 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.547964    2557 scope.go:117] "RemoveContainer" containerID="ca3d4a057acac34e423b60137d2653f27fc26e012775e11c660c909f36282ea1"
Sep 05 02:34:00.558063 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.558041    2557 scope.go:117] "RemoveContainer" containerID="ca3d4a057acac34e423b60137d2653f27fc26e012775e11c660c909f36282ea1"
Sep 05 02:34:00.558391 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:34:00.558351    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3d4a057acac34e423b60137d2653f27fc26e012775e11c660c909f36282ea1\": container with ID starting with ca3d4a057acac34e423b60137d2653f27fc26e012775e11c660c909f36282ea1 not found: ID does not exist" containerID="ca3d4a057acac34e423b60137d2653f27fc26e012775e11c660c909f36282ea1"
Sep 05 02:34:00.558488 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.558398    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3d4a057acac34e423b60137d2653f27fc26e012775e11c660c909f36282ea1"} err="failed to get container status \"ca3d4a057acac34e423b60137d2653f27fc26e012775e11c660c909f36282ea1\": rpc error: code = NotFound desc = could not find container \"ca3d4a057acac34e423b60137d2653f27fc26e012775e11c660c909f36282ea1\": container with ID starting with ca3d4a057acac34e423b60137d2653f27fc26e012775e11c660c909f36282ea1 not found: ID does not exist"
Sep 05 02:34:00.569641 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.569615    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"]
Sep 05 02:34:00.575603 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.575579    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-rqwkc"]
Sep 05 02:34:00.991264 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:00.991230    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="783a0c6c-c678-429d-a1b2-7aa987eb9248" path="/var/lib/kubelet/pods/783a0c6c-c678-429d-a1b2-7aa987eb9248/volumes"
Sep 05 02:34:05.022238 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:05.022202    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:34:05.117662 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:05.117628    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:34:08.713103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:08.713051    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 05 02:34:10.963517 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:10.963482    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:34:16.731864 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:16.731827    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:34:16.833237 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:16.833204    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:34:18.092162 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.092124    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:34:18.329051 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.329006    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"]
Sep 05 02:34:18.329602 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.329580    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="783a0c6c-c678-429d-a1b2-7aa987eb9248" containerName="main"
Sep 05 02:34:18.329694 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.329606    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="783a0c6c-c678-429d-a1b2-7aa987eb9248" containerName="main"
Sep 05 02:34:18.329731 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.329721    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="783a0c6c-c678-429d-a1b2-7aa987eb9248" containerName="main"
Sep 05 02:34:18.335301 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.335278    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"
Sep 05 02:34:18.339365 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.338264    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 05 02:34:18.339365 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.338583    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 05 02:34:18.339365 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.338659    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 05 02:34:18.339365 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.338994    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-m9bpf\""
Sep 05 02:34:18.339365 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.339141    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-t5vw5\""
Sep 05 02:34:18.342673 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.342613    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"]
Sep 05 02:34:18.359477 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.359428    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:34:18.453512 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.453463    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485928e9-ea87-4301-baed-0989476d3baa-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d\" (UID: \"485928e9-ea87-4301-baed-0989476d3baa\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"
Sep 05 02:34:18.453676 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.453622    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7cx7\" (UniqueName: \"kubernetes.io/projected/485928e9-ea87-4301-baed-0989476d3baa-kube-api-access-v7cx7\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d\" (UID: \"485928e9-ea87-4301-baed-0989476d3baa\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"
Sep 05 02:34:18.554328 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.554292    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7cx7\" (UniqueName: \"kubernetes.io/projected/485928e9-ea87-4301-baed-0989476d3baa-kube-api-access-v7cx7\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d\" (UID: \"485928e9-ea87-4301-baed-0989476d3baa\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"
Sep 05 02:34:18.554538 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.554375    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485928e9-ea87-4301-baed-0989476d3baa-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d\" (UID: \"485928e9-ea87-4301-baed-0989476d3baa\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"
Sep 05 02:34:18.557064 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.557028    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485928e9-ea87-4301-baed-0989476d3baa-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d\" (UID: \"485928e9-ea87-4301-baed-0989476d3baa\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"
Sep 05 02:34:18.562259 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.562230    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7cx7\" (UniqueName: \"kubernetes.io/projected/485928e9-ea87-4301-baed-0989476d3baa-kube-api-access-v7cx7\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d\" (UID: \"485928e9-ea87-4301-baed-0989476d3baa\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"
Sep 05 02:34:18.649652 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.649559    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"
Sep 05 02:34:18.713067 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.713023    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 05 02:34:18.929659 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:18.929626    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"]
Sep 05 02:34:18.931133 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:34:18.931105    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485928e9_ea87_4301_baed_0989476d3baa.slice/crio-28d00b89e90346f13595e7e61522bda985934aaf122e08eced9a60fca614a13b WatchSource:0}: Error finding container 28d00b89e90346f13595e7e61522bda985934aaf122e08eced9a60fca614a13b: Status 404 returned error can't find the container with id 28d00b89e90346f13595e7e61522bda985934aaf122e08eced9a60fca614a13b
Sep 05 02:34:19.655484 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:19.655429    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d" event={"ID":"485928e9-ea87-4301-baed-0989476d3baa","Type":"ContainerStarted","Data":"d7af52f8224d6e8afe3f97dd420a357baf1413ef2c039f4b205192b48b3e7387"}
Sep 05 02:34:19.655956 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:19.655515    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"
Sep 05 02:34:19.655956 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:19.655534    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d" event={"ID":"485928e9-ea87-4301-baed-0989476d3baa","Type":"ContainerStarted","Data":"28d00b89e90346f13595e7e61522bda985934aaf122e08eced9a60fca614a13b"}
Sep 05 02:34:19.672852 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:19.672800    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d" podStartSLOduration=1.6727849 podStartE2EDuration="1.6727849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:34:19.67092153 +0000 UTC m=+963.226077840" watchObservedRunningTime="2026-09-05 02:34:19.6727849 +0000 UTC m=+963.227941228"
Sep 05 02:34:28.723012 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:28.722980    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:34:28.731112 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:28.731090    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:34:29.445227 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:29.445191    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:34:34.517009 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:34.516965    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"]
Sep 05 02:34:34.525707 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:34.525682    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:34:49.540654 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:49.540569    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main" containerID="cri-o://6bf034201807fbbc6598869d2f15d6d8dbc8b612d76eedb0967413d68ac84e3c" gracePeriod=45
Sep 05 02:34:49.724083 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:49.724040    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 05 02:34:50.290701 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.290675    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:34:50.448231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.448198    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-home\") pod \"279be1f8-14e4-43af-b769-30b1106a5475\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") "
Sep 05 02:34:50.448231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.448238    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtf7j\" (UniqueName: \"kubernetes.io/projected/279be1f8-14e4-43af-b769-30b1106a5475-kube-api-access-rtf7j\") pod \"279be1f8-14e4-43af-b769-30b1106a5475\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") "
Sep 05 02:34:50.448477 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.448281    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/279be1f8-14e4-43af-b769-30b1106a5475-cabundle-cert\") pod \"279be1f8-14e4-43af-b769-30b1106a5475\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") "
Sep 05 02:34:50.448477 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.448302    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-kserve-provision-location\") pod \"279be1f8-14e4-43af-b769-30b1106a5475\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") "
Sep 05 02:34:50.448477 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.448350    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-dshm\") pod \"279be1f8-14e4-43af-b769-30b1106a5475\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") "
Sep 05 02:34:50.448477 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.448431    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279be1f8-14e4-43af-b769-30b1106a5475-tls-certs\") pod \"279be1f8-14e4-43af-b769-30b1106a5475\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") "
Sep 05 02:34:50.448699 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.448504    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-tmp-dir\") pod \"279be1f8-14e4-43af-b769-30b1106a5475\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") "
Sep 05 02:34:50.448699 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.448549    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-model-cache\") pod \"279be1f8-14e4-43af-b769-30b1106a5475\" (UID: \"279be1f8-14e4-43af-b769-30b1106a5475\") "
Sep 05 02:34:50.448836 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.448731    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279be1f8-14e4-43af-b769-30b1106a5475-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "279be1f8-14e4-43af-b769-30b1106a5475" (UID: "279be1f8-14e4-43af-b769-30b1106a5475"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:34:50.449063 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.449010    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-model-cache" (OuterVolumeSpecName: "model-cache") pod "279be1f8-14e4-43af-b769-30b1106a5475" (UID: "279be1f8-14e4-43af-b769-30b1106a5475"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:34:50.449179 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.449070    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-home" (OuterVolumeSpecName: "home") pod "279be1f8-14e4-43af-b769-30b1106a5475" (UID: "279be1f8-14e4-43af-b769-30b1106a5475"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:34:50.450661 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.450635    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279be1f8-14e4-43af-b769-30b1106a5475-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "279be1f8-14e4-43af-b769-30b1106a5475" (UID: "279be1f8-14e4-43af-b769-30b1106a5475"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:34:50.450923 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.450895    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-dshm" (OuterVolumeSpecName: "dshm") pod "279be1f8-14e4-43af-b769-30b1106a5475" (UID: "279be1f8-14e4-43af-b769-30b1106a5475"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:34:50.451031 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.450948    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279be1f8-14e4-43af-b769-30b1106a5475-kube-api-access-rtf7j" (OuterVolumeSpecName: "kube-api-access-rtf7j") pod "279be1f8-14e4-43af-b769-30b1106a5475" (UID: "279be1f8-14e4-43af-b769-30b1106a5475"). InnerVolumeSpecName "kube-api-access-rtf7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:34:50.461412 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.461383    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "279be1f8-14e4-43af-b769-30b1106a5475" (UID: "279be1f8-14e4-43af-b769-30b1106a5475"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:34:50.469314 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.469289    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "279be1f8-14e4-43af-b769-30b1106a5475" (UID: "279be1f8-14e4-43af-b769-30b1106a5475"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:34:50.549615 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.549586    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/279be1f8-14e4-43af-b769-30b1106a5475-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:34:50.549615 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.549612    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:34:50.549615 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.549623    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:34:50.550010 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.549633    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279be1f8-14e4-43af-b769-30b1106a5475-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:34:50.550010 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.549642    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:34:50.550010 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.549650    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:34:50.550010 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.549658    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279be1f8-14e4-43af-b769-30b1106a5475-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:34:50.550010 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.549666    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtf7j\" (UniqueName: \"kubernetes.io/projected/279be1f8-14e4-43af-b769-30b1106a5475-kube-api-access-rtf7j\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:34:50.662962 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.662928    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"
Sep 05 02:34:50.836944 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.836905    2557 generic.go:356] "Generic (PLEG): container finished" podID="279be1f8-14e4-43af-b769-30b1106a5475" containerID="6bf034201807fbbc6598869d2f15d6d8dbc8b612d76eedb0967413d68ac84e3c" exitCode=0
Sep 05 02:34:50.837127 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.836957    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"
Sep 05 02:34:50.837127 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.836989    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" event={"ID":"279be1f8-14e4-43af-b769-30b1106a5475","Type":"ContainerDied","Data":"6bf034201807fbbc6598869d2f15d6d8dbc8b612d76eedb0967413d68ac84e3c"}
Sep 05 02:34:50.837127 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.837043    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g" event={"ID":"279be1f8-14e4-43af-b769-30b1106a5475","Type":"ContainerDied","Data":"a3ac7564ebdf603dfb2b302d476dbd673e32346258b49f269114bf4e470de17b"}
Sep 05 02:34:50.837127 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.837066    2557 scope.go:117] "RemoveContainer" containerID="6bf034201807fbbc6598869d2f15d6d8dbc8b612d76eedb0967413d68ac84e3c"
Sep 05 02:34:50.846767 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.846743    2557 scope.go:117] "RemoveContainer" containerID="4ba9cf640c54fc1aa76c8d73f163f570e2bff291b283e0eb814cc577ba45c961"
Sep 05 02:34:50.854886 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.854863    2557 scope.go:117] "RemoveContainer" containerID="6bf034201807fbbc6598869d2f15d6d8dbc8b612d76eedb0967413d68ac84e3c"
Sep 05 02:34:50.855172 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:34:50.855153    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf034201807fbbc6598869d2f15d6d8dbc8b612d76eedb0967413d68ac84e3c\": container with ID starting with 6bf034201807fbbc6598869d2f15d6d8dbc8b612d76eedb0967413d68ac84e3c not found: ID does not exist" containerID="6bf034201807fbbc6598869d2f15d6d8dbc8b612d76eedb0967413d68ac84e3c"
Sep 05 02:34:50.855222 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.855182    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf034201807fbbc6598869d2f15d6d8dbc8b612d76eedb0967413d68ac84e3c"} err="failed to get container status \"6bf034201807fbbc6598869d2f15d6d8dbc8b612d76eedb0967413d68ac84e3c\": rpc error: code = NotFound desc = could not find container \"6bf034201807fbbc6598869d2f15d6d8dbc8b612d76eedb0967413d68ac84e3c\": container with ID starting with 6bf034201807fbbc6598869d2f15d6d8dbc8b612d76eedb0967413d68ac84e3c not found: ID does not exist"
Sep 05 02:34:50.855222 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.855199    2557 scope.go:117] "RemoveContainer" containerID="4ba9cf640c54fc1aa76c8d73f163f570e2bff291b283e0eb814cc577ba45c961"
Sep 05 02:34:50.855472 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:34:50.855454    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba9cf640c54fc1aa76c8d73f163f570e2bff291b283e0eb814cc577ba45c961\": container with ID starting with 4ba9cf640c54fc1aa76c8d73f163f570e2bff291b283e0eb814cc577ba45c961 not found: ID does not exist" containerID="4ba9cf640c54fc1aa76c8d73f163f570e2bff291b283e0eb814cc577ba45c961"
Sep 05 02:34:50.855526 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.855476    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba9cf640c54fc1aa76c8d73f163f570e2bff291b283e0eb814cc577ba45c961"} err="failed to get container status \"4ba9cf640c54fc1aa76c8d73f163f570e2bff291b283e0eb814cc577ba45c961\": rpc error: code = NotFound desc = could not find container \"4ba9cf640c54fc1aa76c8d73f163f570e2bff291b283e0eb814cc577ba45c961\": container with ID starting with 4ba9cf640c54fc1aa76c8d73f163f570e2bff291b283e0eb814cc577ba45c961 not found: ID does not exist"
Sep 05 02:34:50.860092 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.860066    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"]
Sep 05 02:34:50.864315 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.864289    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-fb74c456b-hk97g"]
Sep 05 02:34:50.990266 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:50.990230    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="279be1f8-14e4-43af-b769-30b1106a5475" path="/var/lib/kubelet/pods/279be1f8-14e4-43af-b769-30b1106a5475/volumes"
Sep 05 02:34:54.585984 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:54.585942    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:34:54.683763 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:34:54.683731    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:00.623531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:00.623495    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:03.070299 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.070256    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:03.310141 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.310100    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:03.402938 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.402818    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:03.406190 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.406165    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 05 02:35:03.406552 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.406539    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main"
Sep 05 02:35:03.406604 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.406554    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main"
Sep 05 02:35:03.406604 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.406568    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="storage-initializer"
Sep 05 02:35:03.406604 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.406573    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="storage-initializer"
Sep 05 02:35:03.406701 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.406630    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="279be1f8-14e4-43af-b769-30b1106a5475" containerName="main"
Sep 05 02:35:03.409729 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.409708    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.412323 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.412298    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 05 02:35:03.412477 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.412297    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 05 02:35:03.412938 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.412916    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-r2g49\""
Sep 05 02:35:03.413054 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.413008    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 05 02:35:03.413121 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.413068    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 05 02:35:03.413175 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.413068    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-gcxjp\""
Sep 05 02:35:03.413175 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.413132    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:35:03.423931 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.423900    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 05 02:35:03.563964 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.563934    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:03.566099 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.566076    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.566200 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.566121    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.566200 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.566145    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2aa2aac-c629-44c6-9f84-5e577a9659b8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.566200 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.566171    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c76jd\" (UniqueName: \"kubernetes.io/projected/f2aa2aac-c629-44c6-9f84-5e577a9659b8-kube-api-access-c76jd\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.566324 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.566201    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.566324 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.566257    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2aa2aac-c629-44c6-9f84-5e577a9659b8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.566324 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.566277    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.566324 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.566291    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.667512 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.667404    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.667512 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.667484    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2aa2aac-c629-44c6-9f84-5e577a9659b8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.667512 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.667510    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.667812 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.667535    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.667812 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.667618    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.667812 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.667651    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.667812 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.667679    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2aa2aac-c629-44c6-9f84-5e577a9659b8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.667812 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.667706    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c76jd\" (UniqueName: \"kubernetes.io/projected/f2aa2aac-c629-44c6-9f84-5e577a9659b8-kube-api-access-c76jd\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.668096 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.667889    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.668096 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.667943    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.668096 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.667963    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.668096 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.668081    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.668464 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.668425    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2aa2aac-c629-44c6-9f84-5e577a9659b8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.669896 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.669872    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.670088 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.670065    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2aa2aac-c629-44c6-9f84-5e577a9659b8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.675379 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.675356    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c76jd\" (UniqueName: \"kubernetes.io/projected/f2aa2aac-c629-44c6-9f84-5e577a9659b8-kube-api-access-c76jd\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:03.720032 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:03.719994    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:04.000139 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:04.000110    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 05 02:35:04.001683 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:35:04.001650    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2aa2aac_c629_44c6_9f84_5e577a9659b8.slice/crio-62f9f464cce3319883648dea7e4dbec7e53ff32c693301cbe02c8bb094dcc9b6 WatchSource:0}: Error finding container 62f9f464cce3319883648dea7e4dbec7e53ff32c693301cbe02c8bb094dcc9b6: Status 404 returned error can't find the container with id 62f9f464cce3319883648dea7e4dbec7e53ff32c693301cbe02c8bb094dcc9b6
Sep 05 02:35:04.915964 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:04.915919    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f2aa2aac-c629-44c6-9f84-5e577a9659b8","Type":"ContainerStarted","Data":"e399ebe1989dd6442bca1d0345179b702481a2c27f3151a63e984c66e5ef92db"}
Sep 05 02:35:04.916341 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:04.915980    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f2aa2aac-c629-44c6-9f84-5e577a9659b8","Type":"ContainerStarted","Data":"62f9f464cce3319883648dea7e4dbec7e53ff32c693301cbe02c8bb094dcc9b6"}
Sep 05 02:35:06.496966 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:06.496933    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"]
Sep 05 02:35:06.506690 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:06.506656    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:06.617531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:06.617493    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:06.775262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:06.775167    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:06.875558 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:06.875523    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:06.929183 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:06.929150    2557 generic.go:356] "Generic (PLEG): container finished" podID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerID="e399ebe1989dd6442bca1d0345179b702481a2c27f3151a63e984c66e5ef92db" exitCode=0
Sep 05 02:35:06.929355 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:06.929197    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f2aa2aac-c629-44c6-9f84-5e577a9659b8","Type":"ContainerDied","Data":"e399ebe1989dd6442bca1d0345179b702481a2c27f3151a63e984c66e5ef92db"}
Sep 05 02:35:07.936605 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:07.936561    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f2aa2aac-c629-44c6-9f84-5e577a9659b8","Type":"ContainerStarted","Data":"f141a22f007be701b06c8a57111c84b0e9571fd657e4e33fa34f3fd7eea613b8"}
Sep 05 02:35:07.958004 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:07.957944    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.607129271 podStartE2EDuration="4.957929673s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.350800402s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:35:07.956743919 +0000 UTC m=+1011.511900255" watchObservedRunningTime="2026-09-05 02:35:07.957929673 +0000 UTC m=+1011.513085982"
Sep 05 02:35:08.720695 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:08.720658    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:10.663543 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:10.663503    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d" podUID="485928e9-ea87-4301-baed-0989476d3baa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:35:10.696372 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:10.696328    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:13.720205 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:13.720168    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:35:13.721969 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:13.721937    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 05 02:35:20.662942 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:20.662894    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d" podUID="485928e9-ea87-4301-baed-0989476d3baa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:35:21.518731 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:21.518655    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d" podUID="485928e9-ea87-4301-baed-0989476d3baa" containerName="main" containerID="cri-o://d7af52f8224d6e8afe3f97dd420a357baf1413ef2c039f4b205192b48b3e7387" gracePeriod=45
Sep 05 02:35:23.720919 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:23.720878    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 05 02:35:30.663095 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:30.663038    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d" podUID="485928e9-ea87-4301-baed-0989476d3baa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:35:30.663567 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:30.663181    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"
Sep 05 02:35:33.720695 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:33.720639    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 05 02:35:40.662999 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:40.662959    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d" podUID="485928e9-ea87-4301-baed-0989476d3baa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:35:43.720404 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:43.720357    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 05 02:35:50.662744 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:50.662701    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d" podUID="485928e9-ea87-4301-baed-0989476d3baa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:35:51.771707 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:51.771680    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"
Sep 05 02:35:51.912050 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:51.912009    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7cx7\" (UniqueName: \"kubernetes.io/projected/485928e9-ea87-4301-baed-0989476d3baa-kube-api-access-v7cx7\") pod \"485928e9-ea87-4301-baed-0989476d3baa\" (UID: \"485928e9-ea87-4301-baed-0989476d3baa\") "
Sep 05 02:35:51.912249 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:51.912145    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485928e9-ea87-4301-baed-0989476d3baa-tls-certs\") pod \"485928e9-ea87-4301-baed-0989476d3baa\" (UID: \"485928e9-ea87-4301-baed-0989476d3baa\") "
Sep 05 02:35:51.914304 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:51.914272    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/485928e9-ea87-4301-baed-0989476d3baa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "485928e9-ea87-4301-baed-0989476d3baa" (UID: "485928e9-ea87-4301-baed-0989476d3baa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:35:51.914411 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:51.914336    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485928e9-ea87-4301-baed-0989476d3baa-kube-api-access-v7cx7" (OuterVolumeSpecName: "kube-api-access-v7cx7") pod "485928e9-ea87-4301-baed-0989476d3baa" (UID: "485928e9-ea87-4301-baed-0989476d3baa"). InnerVolumeSpecName "kube-api-access-v7cx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:35:52.012796 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:52.012757    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485928e9-ea87-4301-baed-0989476d3baa-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:35:52.012796 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:52.012792    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7cx7\" (UniqueName: \"kubernetes.io/projected/485928e9-ea87-4301-baed-0989476d3baa-kube-api-access-v7cx7\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:35:52.197207 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:52.197122    2557 generic.go:356] "Generic (PLEG): container finished" podID="485928e9-ea87-4301-baed-0989476d3baa" containerID="d7af52f8224d6e8afe3f97dd420a357baf1413ef2c039f4b205192b48b3e7387" exitCode=0
Sep 05 02:35:52.197207 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:52.197165    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"
Sep 05 02:35:52.197382 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:52.197202    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d" event={"ID":"485928e9-ea87-4301-baed-0989476d3baa","Type":"ContainerDied","Data":"d7af52f8224d6e8afe3f97dd420a357baf1413ef2c039f4b205192b48b3e7387"}
Sep 05 02:35:52.197382 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:52.197260    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d" event={"ID":"485928e9-ea87-4301-baed-0989476d3baa","Type":"ContainerDied","Data":"28d00b89e90346f13595e7e61522bda985934aaf122e08eced9a60fca614a13b"}
Sep 05 02:35:52.197382 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:52.197289    2557 scope.go:117] "RemoveContainer" containerID="d7af52f8224d6e8afe3f97dd420a357baf1413ef2c039f4b205192b48b3e7387"
Sep 05 02:35:52.207676 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:52.207651    2557 scope.go:117] "RemoveContainer" containerID="d7af52f8224d6e8afe3f97dd420a357baf1413ef2c039f4b205192b48b3e7387"
Sep 05 02:35:52.207955 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:35:52.207938    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7af52f8224d6e8afe3f97dd420a357baf1413ef2c039f4b205192b48b3e7387\": container with ID starting with d7af52f8224d6e8afe3f97dd420a357baf1413ef2c039f4b205192b48b3e7387 not found: ID does not exist" containerID="d7af52f8224d6e8afe3f97dd420a357baf1413ef2c039f4b205192b48b3e7387"
Sep 05 02:35:52.208005 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:52.207962    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7af52f8224d6e8afe3f97dd420a357baf1413ef2c039f4b205192b48b3e7387"} err="failed to get container status \"d7af52f8224d6e8afe3f97dd420a357baf1413ef2c039f4b205192b48b3e7387\": rpc error: code = NotFound desc = could not find container \"d7af52f8224d6e8afe3f97dd420a357baf1413ef2c039f4b205192b48b3e7387\": container with ID starting with d7af52f8224d6e8afe3f97dd420a357baf1413ef2c039f4b205192b48b3e7387 not found: ID does not exist"
Sep 05 02:35:52.218263 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:52.218232    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"]
Sep 05 02:35:52.222035 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:52.222009    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rzc7d"]
Sep 05 02:35:52.993159 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:52.993127    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="485928e9-ea87-4301-baed-0989476d3baa" path="/var/lib/kubelet/pods/485928e9-ea87-4301-baed-0989476d3baa/volumes"
Sep 05 02:35:53.720938 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:53.720888    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 05 02:35:57.228373 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.228333    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:57.331372 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.331338    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:57.489892 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.489858    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"]
Sep 05 02:35:57.490514 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.490490    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="485928e9-ea87-4301-baed-0989476d3baa" containerName="main"
Sep 05 02:35:57.490514 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.490516    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="485928e9-ea87-4301-baed-0989476d3baa" containerName="main"
Sep 05 02:35:57.490637 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.490622    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="485928e9-ea87-4301-baed-0989476d3baa" containerName="main"
Sep 05 02:35:57.495385 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.495363    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.498006 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.497942    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 05 02:35:57.498006 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.497975    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 05 02:35:57.498798 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.498778    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 05 02:35:57.498893 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.498808    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-p89nj\""
Sep 05 02:35:57.507324 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.507298    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"]
Sep 05 02:35:57.523066 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.523025    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:57.616304 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.616263    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:35:57.662698 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.662659    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.662864 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.662708    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.662864 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.662797    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.662864 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.662826    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.662979 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.662864    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.662979 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.662883    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271648aa-70e9-496c-8a42-7aa9846e8df5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.662979 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.662899    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t862\" (UniqueName: \"kubernetes.io/projected/271648aa-70e9-496c-8a42-7aa9846e8df5-kube-api-access-8t862\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.662979 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.662939    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.663114 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.662998    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.763798 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.763704    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.763798 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.763742    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.763798 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.763775    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.764080 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.763827    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.764080 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.763844    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.764080 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.763861    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.764080 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.763882    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271648aa-70e9-496c-8a42-7aa9846e8df5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.764080 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.763908    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t862\" (UniqueName: \"kubernetes.io/projected/271648aa-70e9-496c-8a42-7aa9846e8df5-kube-api-access-8t862\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.764080 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.763962    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.764378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.764189    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.764378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.764242    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.764513 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.764386    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.764663 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.764635    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.764850 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.764832    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271648aa-70e9-496c-8a42-7aa9846e8df5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.766773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.766751    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.767100 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.767076    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.772511 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.772485    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.772669 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.772650    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t862\" (UniqueName: \"kubernetes.io/projected/271648aa-70e9-496c-8a42-7aa9846e8df5-kube-api-access-8t862\") pod \"router-gateway-1-openshift-default-5b4d9458bd-dgzdl\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:57.807720 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:57.807685    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:58.088232 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:58.088201    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"]
Sep 05 02:35:58.089866 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:35:58.089824    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271648aa_70e9_496c_8a42_7aa9846e8df5.slice/crio-4bf294db12ecdec2ff6f33d800e8ef3c72711c6b6afd3a2542e4859c65ec778c WatchSource:0}: Error finding container 4bf294db12ecdec2ff6f33d800e8ef3c72711c6b6afd3a2542e4859c65ec778c: Status 404 returned error can't find the container with id 4bf294db12ecdec2ff6f33d800e8ef3c72711c6b6afd3a2542e4859c65ec778c
Sep 05 02:35:58.091973 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:58.091940    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 02:35:58.092076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:58.092059    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 02:35:58.092155 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:58.092107    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 02:35:58.236053 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:58.236019    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl" event={"ID":"271648aa-70e9-496c-8a42-7aa9846e8df5","Type":"ContainerStarted","Data":"e464103db05ad456ba630d4134ffa3fabb096053f930a602cac00183dd0052d9"}
Sep 05 02:35:58.236510 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:58.236092    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl" event={"ID":"271648aa-70e9-496c-8a42-7aa9846e8df5","Type":"ContainerStarted","Data":"4bf294db12ecdec2ff6f33d800e8ef3c72711c6b6afd3a2542e4859c65ec778c"}
Sep 05 02:35:58.259154 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:58.259043    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl" podStartSLOduration=1.259020958 podStartE2EDuration="1.259020958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:35:58.256098911 +0000 UTC m=+1061.811255220" watchObservedRunningTime="2026-09-05 02:35:58.259020958 +0000 UTC m=+1061.814177267"
Sep 05 02:35:58.808165 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:58.808125    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:35:59.814669 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:35:59.814636    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:36:00.248124 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:00.248089    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:36:00.249200 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:00.249179    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:36:01.049411 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:01.049373    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:03.445997 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:03.445951    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:03.721323 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:03.721225    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 05 02:36:06.604040 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.604000    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"]
Sep 05 02:36:06.607851 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.607832    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.611010 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.610978    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 05 02:36:06.611171 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.610983    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-pmzd2\""
Sep 05 02:36:06.618896 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.618870    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"]
Sep 05 02:36:06.645117 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.645087    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7g8w\" (UniqueName: \"kubernetes.io/projected/8d890f87-f9b4-4b62-8d80-22be65ea71d4-kube-api-access-f7g8w\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.645267 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.645135    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d890f87-f9b4-4b62-8d80-22be65ea71d4-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.645267 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.645194    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.645267 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.645217    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.645267 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.645240    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.645267 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.645254    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.708600 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.708555    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:06.746040 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.746003    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7g8w\" (UniqueName: \"kubernetes.io/projected/8d890f87-f9b4-4b62-8d80-22be65ea71d4-kube-api-access-f7g8w\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.746235 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.746058    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d890f87-f9b4-4b62-8d80-22be65ea71d4-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.746235 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.746088    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.746235 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.746113    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.746235 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.746142    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.746235 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.746163    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.746670 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.746634    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.746808 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.746699    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.746808 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.746757    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.748579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.748558    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.748767 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.748751    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d890f87-f9b4-4b62-8d80-22be65ea71d4-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.754243 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.754216    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7g8w\" (UniqueName: \"kubernetes.io/projected/8d890f87-f9b4-4b62-8d80-22be65ea71d4-kube-api-access-f7g8w\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:06.919339 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:06.919248    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:07.199474 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:07.199431    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"]
Sep 05 02:36:07.201590 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:36:07.201561    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d890f87_f9b4_4b62_8d80_22be65ea71d4.slice/crio-56c74d2be10897a06e205c070f849269771c6226992f493631a089ce7ac9cc6d WatchSource:0}: Error finding container 56c74d2be10897a06e205c070f849269771c6226992f493631a089ce7ac9cc6d: Status 404 returned error can't find the container with id 56c74d2be10897a06e205c070f849269771c6226992f493631a089ce7ac9cc6d
Sep 05 02:36:07.204036 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:07.204020    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:36:07.297720 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:07.297677    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww" event={"ID":"8d890f87-f9b4-4b62-8d80-22be65ea71d4","Type":"ContainerStarted","Data":"56c74d2be10897a06e205c070f849269771c6226992f493631a089ce7ac9cc6d"}
Sep 05 02:36:09.494600 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.494559    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:09.566042 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.565952    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:09.645289 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.645194    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:09.891353 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.891306    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"]
Sep 05 02:36:09.895096 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.895074    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:09.897536 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.897510    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 05 02:36:09.897677 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.897635    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 05 02:36:09.897757 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.897697    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-rkpgk\""
Sep 05 02:36:09.898167 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.898077    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 05 02:36:09.907928 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.907903    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"]
Sep 05 02:36:09.985630 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.985595    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:09.985816 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.985642    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:09.985816 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.985677    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:09.985816 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.985707    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:09.985816 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.985756    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:09.985816 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.985781    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpvbl\" (UniqueName: \"kubernetes.io/projected/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-kube-api-access-qpvbl\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:09.986131 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.985926    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:09.986131 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.985959    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:09.986131 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:09.985994    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.018318 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.018282    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:10.086587 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.086545    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.086784 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.086620    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.086784 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.086651    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.086784 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.086682    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.086784 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.086710    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.086784 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.086766    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.087056 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.086795    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpvbl\" (UniqueName: \"kubernetes.io/projected/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-kube-api-access-qpvbl\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.087056 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.086898    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.087056 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.086926    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.087056 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.086997    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.087056 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.087046    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.087471 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.087431    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.087589 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.087505    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.087795 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.087765    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.089567 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.089538    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.089722 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.089661    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.094647 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.094626    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.094864 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.094843    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpvbl\" (UniqueName: \"kubernetes.io/projected/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-kube-api-access-qpvbl\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5kvf7\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.118244 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.118217    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:10.208764 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.208676    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:10.321699 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.321658    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww" event={"ID":"8d890f87-f9b4-4b62-8d80-22be65ea71d4","Type":"ContainerStarted","Data":"135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d"}
Sep 05 02:36:10.346981 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.346907    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww" podStartSLOduration=2.288446963 podStartE2EDuration="4.346872104s" totalImagesPullingTime="2.058425141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:36:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:36:10.343699941 +0000 UTC m=+1073.898856251" watchObservedRunningTime="2026-09-05 02:36:10.346872104 +0000 UTC m=+1073.902028413"
Sep 05 02:36:10.521690 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.521662    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"]
Sep 05 02:36:10.523643 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:36:10.523615    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd1a0c7_32a2_4cae_9e27_89238d96ff59.slice/crio-480902b627f1e76a87c5565d9c6b8afcca616b9aa2842b4b13d92484e7b9ac1f WatchSource:0}: Error finding container 480902b627f1e76a87c5565d9c6b8afcca616b9aa2842b4b13d92484e7b9ac1f: Status 404 returned error can't find the container with id 480902b627f1e76a87c5565d9c6b8afcca616b9aa2842b4b13d92484e7b9ac1f
Sep 05 02:36:10.526031 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.526000    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 02:36:10.526091 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.526066    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 02:36:10.526129 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.526092    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 02:36:10.920020 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:10.919976    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:11.329224 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:11.329183    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7" event={"ID":"5cd1a0c7-32a2-4cae-9e27-89238d96ff59","Type":"ContainerStarted","Data":"16212d7988c51455cdcda9fe0d65fa0334b954250821712dea7e3886ed9c62fb"}
Sep 05 02:36:11.329390 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:11.329252    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7" event={"ID":"5cd1a0c7-32a2-4cae-9e27-89238d96ff59","Type":"ContainerStarted","Data":"480902b627f1e76a87c5565d9c6b8afcca616b9aa2842b4b13d92484e7b9ac1f"}
Sep 05 02:36:11.350794 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:11.350717    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7" podStartSLOduration=2.350695429 podStartE2EDuration="2.350695429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:36:11.349306727 +0000 UTC m=+1074.904463026" watchObservedRunningTime="2026-09-05 02:36:11.350695429 +0000 UTC m=+1074.905851739"
Sep 05 02:36:12.209472 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:12.209402    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:12.214670 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:12.214638    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:12.335593 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:12.335559    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:12.336161 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:12.336138    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:12.972970 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:12.972927    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:13.721391 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:13.721345    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 05 02:36:14.485986 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.485951    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"]
Sep 05 02:36:14.496226 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:36:14.496187    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:36:14.496226 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:36:14.496226 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:36:14.496226 ip-10-0-143-128 kubenswrapper[2557]:  > containerID="135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d" cmd=["/bin/sleep","15"]
Sep 05 02:36:14.496516 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.496245    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww" message="" err=<
Sep 05 02:36:14.496516 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:36:14.496516 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:36:14.496516 ip-10-0-143-128 kubenswrapper[2557]:  >
Sep 05 02:36:14.496516 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:36:14.496261    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:36:14.496516 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:36:14.496516 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:36:14.496516 ip-10-0-143-128 kubenswrapper[2557]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww" podUID="8d890f87-f9b4-4b62-8d80-22be65ea71d4" containerName="main" containerID="cri-o://135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d"
Sep 05 02:36:14.496516 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.496299    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww" podUID="8d890f87-f9b4-4b62-8d80-22be65ea71d4" containerName="main" containerID="cri-o://135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d" gracePeriod=60
Sep 05 02:36:14.512290 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.512254    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:14.624274 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.624241    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"]
Sep 05 02:36:14.624660 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.624477    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl" podUID="271648aa-70e9-496c-8a42-7aa9846e8df5" containerName="istio-proxy" containerID="cri-o://e464103db05ad456ba630d4134ffa3fabb096053f930a602cac00183dd0052d9" gracePeriod=30
Sep 05 02:36:14.717725 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.717676    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:14.758794 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.758766    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:14.826775 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.826739    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-home\") pod \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") "
Sep 05 02:36:14.826775 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.826781    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-tmp-dir\") pod \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") "
Sep 05 02:36:14.827020 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.826802    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-model-cache\") pod \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") "
Sep 05 02:36:14.827020 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.826840    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d890f87-f9b4-4b62-8d80-22be65ea71d4-tls-certs\") pod \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") "
Sep 05 02:36:14.827020 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.826870    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-dshm\") pod \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") "
Sep 05 02:36:14.827020 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.826921    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7g8w\" (UniqueName: \"kubernetes.io/projected/8d890f87-f9b4-4b62-8d80-22be65ea71d4-kube-api-access-f7g8w\") pod \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\" (UID: \"8d890f87-f9b4-4b62-8d80-22be65ea71d4\") "
Sep 05 02:36:14.827231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.827087    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-home" (OuterVolumeSpecName: "home") pod "8d890f87-f9b4-4b62-8d80-22be65ea71d4" (UID: "8d890f87-f9b4-4b62-8d80-22be65ea71d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:14.827231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.827103    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d890f87-f9b4-4b62-8d80-22be65ea71d4" (UID: "8d890f87-f9b4-4b62-8d80-22be65ea71d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:14.827231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.827118    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d890f87-f9b4-4b62-8d80-22be65ea71d4" (UID: "8d890f87-f9b4-4b62-8d80-22be65ea71d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:14.827389 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.827303    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:14.827389 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.827322    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:14.827389 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.827336    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:14.829296 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.829269    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-dshm" (OuterVolumeSpecName: "dshm") pod "8d890f87-f9b4-4b62-8d80-22be65ea71d4" (UID: "8d890f87-f9b4-4b62-8d80-22be65ea71d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:14.829477 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.829416    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d890f87-f9b4-4b62-8d80-22be65ea71d4-kube-api-access-f7g8w" (OuterVolumeSpecName: "kube-api-access-f7g8w") pod "8d890f87-f9b4-4b62-8d80-22be65ea71d4" (UID: "8d890f87-f9b4-4b62-8d80-22be65ea71d4"). InnerVolumeSpecName "kube-api-access-f7g8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:36:14.829606 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.829494    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d890f87-f9b4-4b62-8d80-22be65ea71d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d890f87-f9b4-4b62-8d80-22be65ea71d4" (UID: "8d890f87-f9b4-4b62-8d80-22be65ea71d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:36:14.928833 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.928786    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d890f87-f9b4-4b62-8d80-22be65ea71d4-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:14.928833 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.928831    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7g8w\" (UniqueName: \"kubernetes.io/projected/8d890f87-f9b4-4b62-8d80-22be65ea71d4-kube-api-access-f7g8w\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:14.929100 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:14.928848    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d890f87-f9b4-4b62-8d80-22be65ea71d4-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:15.355454 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:15.355395    2557 generic.go:356] "Generic (PLEG): container finished" podID="8d890f87-f9b4-4b62-8d80-22be65ea71d4" containerID="135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d" exitCode=0
Sep 05 02:36:15.355666 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:15.355454    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"
Sep 05 02:36:15.355666 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:15.355476    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww" event={"ID":"8d890f87-f9b4-4b62-8d80-22be65ea71d4","Type":"ContainerDied","Data":"135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d"}
Sep 05 02:36:15.355666 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:15.355532    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww" event={"ID":"8d890f87-f9b4-4b62-8d80-22be65ea71d4","Type":"ContainerDied","Data":"56c74d2be10897a06e205c070f849269771c6226992f493631a089ce7ac9cc6d"}
Sep 05 02:36:15.355666 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:15.355561    2557 scope.go:117] "RemoveContainer" containerID="135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d"
Sep 05 02:36:15.366381 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:15.366358    2557 scope.go:117] "RemoveContainer" containerID="135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d"
Sep 05 02:36:15.366720 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:36:15.366696    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d\": container with ID starting with 135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d not found: ID does not exist" containerID="135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d"
Sep 05 02:36:15.366835 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:15.366727    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d"} err="failed to get container status \"135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d\": rpc error: code = NotFound desc = could not find container \"135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d\": container with ID starting with 135320a405623cf11323a0733b9df41ae0f7b1bf2d8dde6a03600e5f16d8944d not found: ID does not exist"
Sep 05 02:36:15.373728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:15.373701    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"]
Sep 05 02:36:15.377000 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:15.376976    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-l4vww"]
Sep 05 02:36:16.104218 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.103132    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl" podUID="271648aa-70e9-496c-8a42-7aa9846e8df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": read tcp 10.133.0.2:33314->10.133.0.47:15021: read: connection reset by peer"
Sep 05 02:36:16.281689 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.281661    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:36:16.342402 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.342365    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-credential-socket\") pod \"271648aa-70e9-496c-8a42-7aa9846e8df5\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") "
Sep 05 02:36:16.342565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.342416    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-workload-certs\") pod \"271648aa-70e9-496c-8a42-7aa9846e8df5\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") "
Sep 05 02:36:16.342565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.342483    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-podinfo\") pod \"271648aa-70e9-496c-8a42-7aa9846e8df5\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") "
Sep 05 02:36:16.342565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.342519    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-workload-socket\") pod \"271648aa-70e9-496c-8a42-7aa9846e8df5\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") "
Sep 05 02:36:16.342565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.342547    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t862\" (UniqueName: \"kubernetes.io/projected/271648aa-70e9-496c-8a42-7aa9846e8df5-kube-api-access-8t862\") pod \"271648aa-70e9-496c-8a42-7aa9846e8df5\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") "
Sep 05 02:36:16.342792 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.342625    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-token\") pod \"271648aa-70e9-496c-8a42-7aa9846e8df5\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") "
Sep 05 02:36:16.342792 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.342672    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "271648aa-70e9-496c-8a42-7aa9846e8df5" (UID: "271648aa-70e9-496c-8a42-7aa9846e8df5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:16.342792 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.342685    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271648aa-70e9-496c-8a42-7aa9846e8df5-istiod-ca-cert\") pod \"271648aa-70e9-496c-8a42-7aa9846e8df5\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") "
Sep 05 02:36:16.342792 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.342772    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-envoy\") pod \"271648aa-70e9-496c-8a42-7aa9846e8df5\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") "
Sep 05 02:36:16.342987 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.342802    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-data\") pod \"271648aa-70e9-496c-8a42-7aa9846e8df5\" (UID: \"271648aa-70e9-496c-8a42-7aa9846e8df5\") "
Sep 05 02:36:16.343057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.343023    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271648aa-70e9-496c-8a42-7aa9846e8df5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "271648aa-70e9-496c-8a42-7aa9846e8df5" (UID: "271648aa-70e9-496c-8a42-7aa9846e8df5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:36:16.343166 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.343149    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271648aa-70e9-496c-8a42-7aa9846e8df5-istiod-ca-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:16.343229 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.343172    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-credential-socket\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:16.343318 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.343295    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "271648aa-70e9-496c-8a42-7aa9846e8df5" (UID: "271648aa-70e9-496c-8a42-7aa9846e8df5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:16.343381 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.343334    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-data" (OuterVolumeSpecName: "istio-data") pod "271648aa-70e9-496c-8a42-7aa9846e8df5" (UID: "271648aa-70e9-496c-8a42-7aa9846e8df5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:16.343697 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.343673    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "271648aa-70e9-496c-8a42-7aa9846e8df5" (UID: "271648aa-70e9-496c-8a42-7aa9846e8df5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:16.345403 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.345378    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "271648aa-70e9-496c-8a42-7aa9846e8df5" (UID: "271648aa-70e9-496c-8a42-7aa9846e8df5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 02:36:16.345998 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.345969    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271648aa-70e9-496c-8a42-7aa9846e8df5-kube-api-access-8t862" (OuterVolumeSpecName: "kube-api-access-8t862") pod "271648aa-70e9-496c-8a42-7aa9846e8df5" (UID: "271648aa-70e9-496c-8a42-7aa9846e8df5"). InnerVolumeSpecName "kube-api-access-8t862". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:36:16.346102 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.346018    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "271648aa-70e9-496c-8a42-7aa9846e8df5" (UID: "271648aa-70e9-496c-8a42-7aa9846e8df5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:16.346416 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.346381    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-token" (OuterVolumeSpecName: "istio-token") pod "271648aa-70e9-496c-8a42-7aa9846e8df5" (UID: "271648aa-70e9-496c-8a42-7aa9846e8df5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:36:16.365323 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.365292    2557 generic.go:356] "Generic (PLEG): container finished" podID="271648aa-70e9-496c-8a42-7aa9846e8df5" containerID="e464103db05ad456ba630d4134ffa3fabb096053f930a602cac00183dd0052d9" exitCode=0
Sep 05 02:36:16.365471 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.365344    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl" event={"ID":"271648aa-70e9-496c-8a42-7aa9846e8df5","Type":"ContainerDied","Data":"e464103db05ad456ba630d4134ffa3fabb096053f930a602cac00183dd0052d9"}
Sep 05 02:36:16.365471 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.365404    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl" event={"ID":"271648aa-70e9-496c-8a42-7aa9846e8df5","Type":"ContainerDied","Data":"4bf294db12ecdec2ff6f33d800e8ef3c72711c6b6afd3a2542e4859c65ec778c"}
Sep 05 02:36:16.365471 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.365427    2557 scope.go:117] "RemoveContainer" containerID="e464103db05ad456ba630d4134ffa3fabb096053f930a602cac00183dd0052d9"
Sep 05 02:36:16.365655 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.365563    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"
Sep 05 02:36:16.376878 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.376857    2557 scope.go:117] "RemoveContainer" containerID="e464103db05ad456ba630d4134ffa3fabb096053f930a602cac00183dd0052d9"
Sep 05 02:36:16.377188 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:36:16.377168    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e464103db05ad456ba630d4134ffa3fabb096053f930a602cac00183dd0052d9\": container with ID starting with e464103db05ad456ba630d4134ffa3fabb096053f930a602cac00183dd0052d9 not found: ID does not exist" containerID="e464103db05ad456ba630d4134ffa3fabb096053f930a602cac00183dd0052d9"
Sep 05 02:36:16.377252 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.377198    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e464103db05ad456ba630d4134ffa3fabb096053f930a602cac00183dd0052d9"} err="failed to get container status \"e464103db05ad456ba630d4134ffa3fabb096053f930a602cac00183dd0052d9\": rpc error: code = NotFound desc = could not find container \"e464103db05ad456ba630d4134ffa3fabb096053f930a602cac00183dd0052d9\": container with ID starting with e464103db05ad456ba630d4134ffa3fabb096053f930a602cac00183dd0052d9 not found: ID does not exist"
Sep 05 02:36:16.391607 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.391582    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"]
Sep 05 02:36:16.396901 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.396876    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-dgzdl"]
Sep 05 02:36:16.444186 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.444152    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-token\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:16.444186 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.444185    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-envoy\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:16.444368 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.444196    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-data\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:16.444368 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.444208    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-workload-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:16.444368 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.444220    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271648aa-70e9-496c-8a42-7aa9846e8df5-istio-podinfo\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:16.444368 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.444230    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271648aa-70e9-496c-8a42-7aa9846e8df5-workload-socket\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:16.444368 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.444245    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t862\" (UniqueName: \"kubernetes.io/projected/271648aa-70e9-496c-8a42-7aa9846e8df5-kube-api-access-8t862\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:16.992026 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.991998    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="271648aa-70e9-496c-8a42-7aa9846e8df5" path="/var/lib/kubelet/pods/271648aa-70e9-496c-8a42-7aa9846e8df5/volumes"
Sep 05 02:36:16.992410 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:16.992393    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d890f87-f9b4-4b62-8d80-22be65ea71d4" path="/var/lib/kubelet/pods/8d890f87-f9b4-4b62-8d80-22be65ea71d4/volumes"
Sep 05 02:36:19.759323 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:19.759288    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:20.939131 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:20.939092    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:23.693712 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.693668    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"]
Sep 05 02:36:23.694265 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.694206    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271648aa-70e9-496c-8a42-7aa9846e8df5" containerName="istio-proxy"
Sep 05 02:36:23.694265 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.694226    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="271648aa-70e9-496c-8a42-7aa9846e8df5" containerName="istio-proxy"
Sep 05 02:36:23.694389 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.694273    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d890f87-f9b4-4b62-8d80-22be65ea71d4" containerName="main"
Sep 05 02:36:23.694389 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.694283    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d890f87-f9b4-4b62-8d80-22be65ea71d4" containerName="main"
Sep 05 02:36:23.694389 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.694371    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="271648aa-70e9-496c-8a42-7aa9846e8df5" containerName="istio-proxy"
Sep 05 02:36:23.694389 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.694382    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d890f87-f9b4-4b62-8d80-22be65ea71d4" containerName="main"
Sep 05 02:36:23.702542 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.702515    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.705867 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.705842    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-zhnjl\""
Sep 05 02:36:23.706009 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.705918    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 05 02:36:23.712457 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.712408    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"]
Sep 05 02:36:23.720746 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.720705    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 05 02:36:23.722708 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.722679    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:23.807616 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.807569    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.807616 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.807612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzb8g\" (UniqueName: \"kubernetes.io/projected/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-kube-api-access-qzb8g\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.807893 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.807709    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.807893 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.807751    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.807893 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.807794    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.807893 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.807809    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.908384 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.908347    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.908604 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.908393    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.908604 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.908513    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.908761 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.908625    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.908761 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.908656    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzb8g\" (UniqueName: \"kubernetes.io/projected/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-kube-api-access-qzb8g\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.908761 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.908716    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.909018 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.908995    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.909122 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.909015    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.909122 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.909065    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.910759 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.910731    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.911005 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.910987    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:23.916340 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:23.916313    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzb8g\" (UniqueName: \"kubernetes.io/projected/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-kube-api-access-qzb8g\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:24.017841 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:24.017730    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:24.299317 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:24.299288    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"]
Sep 05 02:36:24.300848 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:36:24.300819    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec951b8_21b9_4138_8b1f_c6bf92463a5f.slice/crio-7cfd2fe6aed38cffef851cf59c2dd3bbb6ed42a5b30d9f2a411b7e7f6c20a85c WatchSource:0}: Error finding container 7cfd2fe6aed38cffef851cf59c2dd3bbb6ed42a5b30d9f2a411b7e7f6c20a85c: Status 404 returned error can't find the container with id 7cfd2fe6aed38cffef851cf59c2dd3bbb6ed42a5b30d9f2a411b7e7f6c20a85c
Sep 05 02:36:24.411057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:24.411010    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69" event={"ID":"2ec951b8-21b9-4138-8b1f-c6bf92463a5f","Type":"ContainerStarted","Data":"60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d"}
Sep 05 02:36:24.411234 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:24.411087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69" event={"ID":"2ec951b8-21b9-4138-8b1f-c6bf92463a5f","Type":"ContainerStarted","Data":"7cfd2fe6aed38cffef851cf59c2dd3bbb6ed42a5b30d9f2a411b7e7f6c20a85c"}
Sep 05 02:36:24.430033 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:24.429975    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69" podStartSLOduration=1.429959692 podStartE2EDuration="1.429959692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:36:24.427505675 +0000 UTC m=+1087.982661985" watchObservedRunningTime="2026-09-05 02:36:24.429959692 +0000 UTC m=+1087.985116001"
Sep 05 02:36:25.018031 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:25.017999    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:26.179461 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:26.178460    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:26.756829 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:26.756789    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:26.855385 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:26.855347    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:28.381935 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.381899    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"]
Sep 05 02:36:28.395380 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.395345    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:28.395581 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:36:28.395503    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:36:28.395581 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:36:28.395581 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:36:28.395581 ip-10-0-143-128 kubenswrapper[2557]:  > containerID="60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d" cmd=["/bin/sleep","15"]
Sep 05 02:36:28.395581 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.395553    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69" message="" err=<
Sep 05 02:36:28.395581 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:36:28.395581 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:36:28.395581 ip-10-0-143-128 kubenswrapper[2557]:  >
Sep 05 02:36:28.395581 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:36:28.395569    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:36:28.395581 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:36:28.395581 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:36:28.395581 ip-10-0-143-128 kubenswrapper[2557]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69" podUID="2ec951b8-21b9-4138-8b1f-c6bf92463a5f" containerName="main" containerID="cri-o://60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d"
Sep 05 02:36:28.396187 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.395611    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69" podUID="2ec951b8-21b9-4138-8b1f-c6bf92463a5f" containerName="main" containerID="cri-o://60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d" gracePeriod=60
Sep 05 02:36:28.698663 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.698627    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:28.767565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.766740    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzb8g\" (UniqueName: \"kubernetes.io/projected/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-kube-api-access-qzb8g\") pod \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") "
Sep 05 02:36:28.767565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.766863    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-model-cache\") pod \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") "
Sep 05 02:36:28.767565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.766918    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-tmp-dir\") pod \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") "
Sep 05 02:36:28.767565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.766960    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-dshm\") pod \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") "
Sep 05 02:36:28.767565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.767003    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-tls-certs\") pod \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") "
Sep 05 02:36:28.767565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.767032    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-home\") pod \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\" (UID: \"2ec951b8-21b9-4138-8b1f-c6bf92463a5f\") "
Sep 05 02:36:28.768414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.768082    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-home" (OuterVolumeSpecName: "home") pod "2ec951b8-21b9-4138-8b1f-c6bf92463a5f" (UID: "2ec951b8-21b9-4138-8b1f-c6bf92463a5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:28.769149 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.768910    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ec951b8-21b9-4138-8b1f-c6bf92463a5f" (UID: "2ec951b8-21b9-4138-8b1f-c6bf92463a5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:28.769497 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.769284    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ec951b8-21b9-4138-8b1f-c6bf92463a5f" (UID: "2ec951b8-21b9-4138-8b1f-c6bf92463a5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:28.775012 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.774965    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-kube-api-access-qzb8g" (OuterVolumeSpecName: "kube-api-access-qzb8g") pod "2ec951b8-21b9-4138-8b1f-c6bf92463a5f" (UID: "2ec951b8-21b9-4138-8b1f-c6bf92463a5f"). InnerVolumeSpecName "kube-api-access-qzb8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:36:28.775796 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.775760    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ec951b8-21b9-4138-8b1f-c6bf92463a5f" (UID: "2ec951b8-21b9-4138-8b1f-c6bf92463a5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:36:28.776001 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.775965    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-dshm" (OuterVolumeSpecName: "dshm") pod "2ec951b8-21b9-4138-8b1f-c6bf92463a5f" (UID: "2ec951b8-21b9-4138-8b1f-c6bf92463a5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:28.868121 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.867980    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:28.868121 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.868025    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzb8g\" (UniqueName: \"kubernetes.io/projected/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-kube-api-access-qzb8g\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:28.868121 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.868042    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:28.868121 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.868057    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:28.868121 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.868070    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:28.868121 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:28.868082    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec951b8-21b9-4138-8b1f-c6bf92463a5f-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:29.461477 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:29.461421    2557 generic.go:356] "Generic (PLEG): container finished" podID="2ec951b8-21b9-4138-8b1f-c6bf92463a5f" containerID="60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d" exitCode=0
Sep 05 02:36:29.462056 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:29.462029    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69" event={"ID":"2ec951b8-21b9-4138-8b1f-c6bf92463a5f","Type":"ContainerDied","Data":"60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d"}
Sep 05 02:36:29.462210 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:29.462195    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69" event={"ID":"2ec951b8-21b9-4138-8b1f-c6bf92463a5f","Type":"ContainerDied","Data":"7cfd2fe6aed38cffef851cf59c2dd3bbb6ed42a5b30d9f2a411b7e7f6c20a85c"}
Sep 05 02:36:29.462320 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:29.462307    2557 scope.go:117] "RemoveContainer" containerID="60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d"
Sep 05 02:36:29.462603 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:29.462586    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"
Sep 05 02:36:29.477529 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:29.477383    2557 scope.go:117] "RemoveContainer" containerID="60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d"
Sep 05 02:36:29.477868 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:36:29.477842    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d\": container with ID starting with 60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d not found: ID does not exist" containerID="60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d"
Sep 05 02:36:29.477944 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:29.477877    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d"} err="failed to get container status \"60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d\": rpc error: code = NotFound desc = could not find container \"60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d\": container with ID starting with 60dece527eb30302a4cdf7787f0ecd8ed97670fd9bb470c816a2bbbb2c50902d not found: ID does not exist"
Sep 05 02:36:29.481000 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:29.480972    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"]
Sep 05 02:36:29.485294 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:29.485262    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfczs69"]
Sep 05 02:36:30.992852 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:30.992813    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ec951b8-21b9-4138-8b1f-c6bf92463a5f" path="/var/lib/kubelet/pods/2ec951b8-21b9-4138-8b1f-c6bf92463a5f/volumes"
Sep 05 02:36:31.738427 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:31.738377    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"]
Sep 05 02:36:31.738707 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:31.738671    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7" podUID="5cd1a0c7-32a2-4cae-9e27-89238d96ff59" containerName="istio-proxy" containerID="cri-o://16212d7988c51455cdcda9fe0d65fa0334b954250821712dea7e3886ed9c62fb" gracePeriod=30
Sep 05 02:36:31.787119 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:31.787086    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:33.290391 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.290323    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:33.410122 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410011    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-envoy\") pod \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") "
Sep 05 02:36:33.410122 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410056    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-token\") pod \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") "
Sep 05 02:36:33.410122 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410095    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-workload-socket\") pod \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") "
Sep 05 02:36:33.410122 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410124    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-workload-certs\") pod \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") "
Sep 05 02:36:33.410925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410204    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-data\") pod \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") "
Sep 05 02:36:33.410925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410288    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpvbl\" (UniqueName: \"kubernetes.io/projected/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-kube-api-access-qpvbl\") pod \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") "
Sep 05 02:36:33.410925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410335    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-podinfo\") pod \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") "
Sep 05 02:36:33.410925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410391    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-credential-socket\") pod \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") "
Sep 05 02:36:33.410925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410394    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5cd1a0c7-32a2-4cae-9e27-89238d96ff59" (UID: "5cd1a0c7-32a2-4cae-9e27-89238d96ff59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:33.410925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410420    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istiod-ca-cert\") pod \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\" (UID: \"5cd1a0c7-32a2-4cae-9e27-89238d96ff59\") "
Sep 05 02:36:33.410925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410482    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5cd1a0c7-32a2-4cae-9e27-89238d96ff59" (UID: "5cd1a0c7-32a2-4cae-9e27-89238d96ff59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:33.410925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410554    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-data" (OuterVolumeSpecName: "istio-data") pod "5cd1a0c7-32a2-4cae-9e27-89238d96ff59" (UID: "5cd1a0c7-32a2-4cae-9e27-89238d96ff59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:33.410925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410652    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5cd1a0c7-32a2-4cae-9e27-89238d96ff59" (UID: "5cd1a0c7-32a2-4cae-9e27-89238d96ff59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:33.410925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410802    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5cd1a0c7-32a2-4cae-9e27-89238d96ff59" (UID: "5cd1a0c7-32a2-4cae-9e27-89238d96ff59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:36:33.410925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410809    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-data\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:33.410925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410827    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-credential-socket\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:33.410925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410840    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-workload-socket\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:33.410925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.410849    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-workload-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:33.412488 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.412460    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5cd1a0c7-32a2-4cae-9e27-89238d96ff59" (UID: "5cd1a0c7-32a2-4cae-9e27-89238d96ff59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:36:33.412488 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.412460    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5cd1a0c7-32a2-4cae-9e27-89238d96ff59" (UID: "5cd1a0c7-32a2-4cae-9e27-89238d96ff59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 02:36:33.412704 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.412680    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-kube-api-access-qpvbl" (OuterVolumeSpecName: "kube-api-access-qpvbl") pod "5cd1a0c7-32a2-4cae-9e27-89238d96ff59" (UID: "5cd1a0c7-32a2-4cae-9e27-89238d96ff59"). InnerVolumeSpecName "kube-api-access-qpvbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:36:33.412837 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.412819    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-token" (OuterVolumeSpecName: "istio-token") pod "5cd1a0c7-32a2-4cae-9e27-89238d96ff59" (UID: "5cd1a0c7-32a2-4cae-9e27-89238d96ff59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:36:33.489381 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.489337    2557 generic.go:356] "Generic (PLEG): container finished" podID="5cd1a0c7-32a2-4cae-9e27-89238d96ff59" containerID="16212d7988c51455cdcda9fe0d65fa0334b954250821712dea7e3886ed9c62fb" exitCode=0
Sep 05 02:36:33.489381 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.489385    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"
Sep 05 02:36:33.489674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.489421    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7" event={"ID":"5cd1a0c7-32a2-4cae-9e27-89238d96ff59","Type":"ContainerDied","Data":"16212d7988c51455cdcda9fe0d65fa0334b954250821712dea7e3886ed9c62fb"}
Sep 05 02:36:33.489674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.489496    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7" event={"ID":"5cd1a0c7-32a2-4cae-9e27-89238d96ff59","Type":"ContainerDied","Data":"480902b627f1e76a87c5565d9c6b8afcca616b9aa2842b4b13d92484e7b9ac1f"}
Sep 05 02:36:33.489674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.489522    2557 scope.go:117] "RemoveContainer" containerID="16212d7988c51455cdcda9fe0d65fa0334b954250821712dea7e3886ed9c62fb"
Sep 05 02:36:33.499088 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.499055    2557 scope.go:117] "RemoveContainer" containerID="16212d7988c51455cdcda9fe0d65fa0334b954250821712dea7e3886ed9c62fb"
Sep 05 02:36:33.499397 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:36:33.499372    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16212d7988c51455cdcda9fe0d65fa0334b954250821712dea7e3886ed9c62fb\": container with ID starting with 16212d7988c51455cdcda9fe0d65fa0334b954250821712dea7e3886ed9c62fb not found: ID does not exist" containerID="16212d7988c51455cdcda9fe0d65fa0334b954250821712dea7e3886ed9c62fb"
Sep 05 02:36:33.499549 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.499402    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16212d7988c51455cdcda9fe0d65fa0334b954250821712dea7e3886ed9c62fb"} err="failed to get container status \"16212d7988c51455cdcda9fe0d65fa0334b954250821712dea7e3886ed9c62fb\": rpc error: code = NotFound desc = could not find container \"16212d7988c51455cdcda9fe0d65fa0334b954250821712dea7e3886ed9c62fb\": container with ID starting with 16212d7988c51455cdcda9fe0d65fa0334b954250821712dea7e3886ed9c62fb not found: ID does not exist"
Sep 05 02:36:33.510998 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.510948    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"]
Sep 05 02:36:33.511315 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.511292    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpvbl\" (UniqueName: \"kubernetes.io/projected/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-kube-api-access-qpvbl\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:33.511315 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.511315    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-podinfo\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:33.511502 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.511325    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istiod-ca-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:33.511502 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.511335    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-envoy\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:33.511502 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.511343    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cd1a0c7-32a2-4cae-9e27-89238d96ff59-istio-token\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:36:33.517097 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.517068    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5kvf7"]
Sep 05 02:36:33.721877 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:33.721762    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 05 02:36:34.818625 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:34.818586    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:34.917565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:34.917531    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:34.992496 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:34.992464    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cd1a0c7-32a2-4cae-9e27-89238d96ff59" path="/var/lib/kubelet/pods/5cd1a0c7-32a2-4cae-9e27-89238d96ff59/volumes"
Sep 05 02:36:37.331204 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:37.331163    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:38.011082 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:38.011037    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:43.545537 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:43.545503    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:43.730024 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:43.729987    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:36:43.737469 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:43.737424    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:36:46.993227 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:46.993189    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"]
Sep 05 02:36:46.993664 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:46.993543    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ec951b8-21b9-4138-8b1f-c6bf92463a5f" containerName="main"
Sep 05 02:36:46.993664 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:46.993555    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ec951b8-21b9-4138-8b1f-c6bf92463a5f" containerName="main"
Sep 05 02:36:46.993664 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:46.993577    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd1a0c7-32a2-4cae-9e27-89238d96ff59" containerName="istio-proxy"
Sep 05 02:36:46.993664 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:46.993582    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd1a0c7-32a2-4cae-9e27-89238d96ff59" containerName="istio-proxy"
Sep 05 02:36:46.993664 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:46.993630    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd1a0c7-32a2-4cae-9e27-89238d96ff59" containerName="istio-proxy"
Sep 05 02:36:46.993664 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:46.993640    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ec951b8-21b9-4138-8b1f-c6bf92463a5f" containerName="main"
Sep 05 02:36:46.996754 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:46.996731    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:46.999090 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:46.999055    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:36:46.999234 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:46.999168    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 05 02:36:46.999234 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:46.999189    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 05 02:36:46.999495 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:46.999478    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-xt5gh\""
Sep 05 02:36:46.999577 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:46.999504    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 05 02:36:46.999641 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:46.999579    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 05 02:36:47.011557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.011532    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"]
Sep 05 02:36:47.018011 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.017989    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:47.137009 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.136977    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-model-cache\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.137009 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.137011    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2eed68-3f0c-4d70-92ca-844a06794fce-tls-certs\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.137223 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.137031    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-kserve-provision-location\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.137223 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.137175    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c2eed68-3f0c-4d70-92ca-844a06794fce-cabundle-cert\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.137330 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.137262    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lspx\" (UniqueName: \"kubernetes.io/projected/8c2eed68-3f0c-4d70-92ca-844a06794fce-kube-api-access-8lspx\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.137330 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.137297    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-tmp-dir\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.137404 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.137337    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-home\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.137404 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.137361    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-dshm\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.237887 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.237856    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-model-cache\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.238061 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.237896    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2eed68-3f0c-4d70-92ca-844a06794fce-tls-certs\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.238061 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.237923    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-kserve-provision-location\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.238061 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.238009    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c2eed68-3f0c-4d70-92ca-844a06794fce-cabundle-cert\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.238257 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.238077    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lspx\" (UniqueName: \"kubernetes.io/projected/8c2eed68-3f0c-4d70-92ca-844a06794fce-kube-api-access-8lspx\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.238257 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.238106    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-tmp-dir\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.238257 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.238141    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-home\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.238257 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.238164    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-dshm\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.238487 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.238347    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-model-cache\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.238841 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.238818    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-kserve-provision-location\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.239075 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.239012    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-tmp-dir\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.239075 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.239052    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-home\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.239413 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.239393    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c2eed68-3f0c-4d70-92ca-844a06794fce-cabundle-cert\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.240637 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.240615    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-dshm\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.240936 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.240920    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2eed68-3f0c-4d70-92ca-844a06794fce-tls-certs\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.247797 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.247723    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lspx\" (UniqueName: \"kubernetes.io/projected/8c2eed68-3f0c-4d70-92ca-844a06794fce-kube-api-access-8lspx\") pod \"auth-enabled-test-kserve-fc9648fc9-jdgdl\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.308414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.308379    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:47.357903 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.357870    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:47.593773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:47.593746    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"]
Sep 05 02:36:47.596246 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:36:47.596209    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2eed68_3f0c_4d70_92ca_844a06794fce.slice/crio-7335e27d4965733fffff7ad87fa766046a1f4a7a72573239c3ff5ca5c54ed4d5 WatchSource:0}: Error finding container 7335e27d4965733fffff7ad87fa766046a1f4a7a72573239c3ff5ca5c54ed4d5: Status 404 returned error can't find the container with id 7335e27d4965733fffff7ad87fa766046a1f4a7a72573239c3ff5ca5c54ed4d5
Sep 05 02:36:48.573782 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:48.573737    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" event={"ID":"8c2eed68-3f0c-4d70-92ca-844a06794fce","Type":"ContainerStarted","Data":"ab17604af58e03b4321709817d8d41fc2e71061de8a59839bdd947e67b75e05a"}
Sep 05 02:36:48.574171 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:48.573811    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" event={"ID":"8c2eed68-3f0c-4d70-92ca-844a06794fce","Type":"ContainerStarted","Data":"7335e27d4965733fffff7ad87fa766046a1f4a7a72573239c3ff5ca5c54ed4d5"}
Sep 05 02:36:50.588137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:50.588098    2557 generic.go:356] "Generic (PLEG): container finished" podID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerID="ab17604af58e03b4321709817d8d41fc2e71061de8a59839bdd947e67b75e05a" exitCode=0
Sep 05 02:36:50.588630 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:50.588169    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" event={"ID":"8c2eed68-3f0c-4d70-92ca-844a06794fce","Type":"ContainerDied","Data":"ab17604af58e03b4321709817d8d41fc2e71061de8a59839bdd947e67b75e05a"}
Sep 05 02:36:51.596823 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:51.596789    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" event={"ID":"8c2eed68-3f0c-4d70-92ca-844a06794fce","Type":"ContainerStarted","Data":"de9b450371fe1c53096b5551795f637b34412b93da4e893a403137fd522b3443"}
Sep 05 02:36:51.625253 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:51.625200    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" podStartSLOduration=3.240938389 podStartE2EDuration="5.625183489s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.3842451s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:36:51.622217185 +0000 UTC m=+1115.177373505" watchObservedRunningTime="2026-09-05 02:36:51.625183489 +0000 UTC m=+1115.180339830"
Sep 05 02:36:52.309519 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:52.309475    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:52.356281 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:52.356237    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:52.453455 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:52.453398    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:36:57.308981 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:57.308931    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:36:57.310860 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:36:57.310830    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 05 02:37:07.309404 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:07.309360    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 05 02:37:15.128776 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:15.128736    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:37:17.309502 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:17.309394    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 05 02:37:17.437184 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:17.437085    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:37:17.453589 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:17.452789    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 05 02:37:17.505976 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:17.505939    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:37:27.309231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:27.309183    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 05 02:37:32.502619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:32.502577    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="main" containerID="cri-o://f141a22f007be701b06c8a57111c84b0e9571fd657e4e33fa34f3fd7eea613b8" gracePeriod=45
Sep 05 02:37:32.731650 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:32.731602    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 05 02:37:33.254109 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.254076    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:37:33.367832 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.367793    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-tmp-dir\") pod \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") "
Sep 05 02:37:33.367998 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.367864    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2aa2aac-c629-44c6-9f84-5e577a9659b8-cabundle-cert\") pod \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") "
Sep 05 02:37:33.367998 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.367906    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2aa2aac-c629-44c6-9f84-5e577a9659b8-tls-certs\") pod \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") "
Sep 05 02:37:33.367998 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.367935    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-model-cache\") pod \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") "
Sep 05 02:37:33.368165 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.368030    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-dshm\") pod \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") "
Sep 05 02:37:33.368165 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.368079    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c76jd\" (UniqueName: \"kubernetes.io/projected/f2aa2aac-c629-44c6-9f84-5e577a9659b8-kube-api-access-c76jd\") pod \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") "
Sep 05 02:37:33.368165 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.368110    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-kserve-provision-location\") pod \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") "
Sep 05 02:37:33.368165 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.368140    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-home\") pod \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\" (UID: \"f2aa2aac-c629-44c6-9f84-5e577a9659b8\") "
Sep 05 02:37:33.368377 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.368210    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2aa2aac-c629-44c6-9f84-5e577a9659b8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2aa2aac-c629-44c6-9f84-5e577a9659b8" (UID: "f2aa2aac-c629-44c6-9f84-5e577a9659b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:37:33.368515 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.368467    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2aa2aac-c629-44c6-9f84-5e577a9659b8-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:37:33.368638 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.368522    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2aa2aac-c629-44c6-9f84-5e577a9659b8" (UID: "f2aa2aac-c629-44c6-9f84-5e577a9659b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:37:33.368908 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.368867    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-home" (OuterVolumeSpecName: "home") pod "f2aa2aac-c629-44c6-9f84-5e577a9659b8" (UID: "f2aa2aac-c629-44c6-9f84-5e577a9659b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:37:33.370361 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.370328    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-dshm" (OuterVolumeSpecName: "dshm") pod "f2aa2aac-c629-44c6-9f84-5e577a9659b8" (UID: "f2aa2aac-c629-44c6-9f84-5e577a9659b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:37:33.370613 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.370577    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2aa2aac-c629-44c6-9f84-5e577a9659b8-kube-api-access-c76jd" (OuterVolumeSpecName: "kube-api-access-c76jd") pod "f2aa2aac-c629-44c6-9f84-5e577a9659b8" (UID: "f2aa2aac-c629-44c6-9f84-5e577a9659b8"). InnerVolumeSpecName "kube-api-access-c76jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:37:33.370709 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.370660    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2aa2aac-c629-44c6-9f84-5e577a9659b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2aa2aac-c629-44c6-9f84-5e577a9659b8" (UID: "f2aa2aac-c629-44c6-9f84-5e577a9659b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:37:33.388847 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.388807    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2aa2aac-c629-44c6-9f84-5e577a9659b8" (UID: "f2aa2aac-c629-44c6-9f84-5e577a9659b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:37:33.391368 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.391345    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2aa2aac-c629-44c6-9f84-5e577a9659b8" (UID: "f2aa2aac-c629-44c6-9f84-5e577a9659b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:37:33.470019 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.469982    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:37:33.470167 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.470025    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c76jd\" (UniqueName: \"kubernetes.io/projected/f2aa2aac-c629-44c6-9f84-5e577a9659b8-kube-api-access-c76jd\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:37:33.470167 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.470042    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:37:33.470167 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.470058    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:37:33.470167 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.470097    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:37:33.470167 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.470111    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2aa2aac-c629-44c6-9f84-5e577a9659b8-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:37:33.470167 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.470123    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2aa2aac-c629-44c6-9f84-5e577a9659b8-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:37:33.850944 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.850906    2557 generic.go:356] "Generic (PLEG): container finished" podID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerID="f141a22f007be701b06c8a57111c84b0e9571fd657e4e33fa34f3fd7eea613b8" exitCode=0
Sep 05 02:37:33.851349 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.850979    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 05 02:37:33.851349 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.850977    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f2aa2aac-c629-44c6-9f84-5e577a9659b8","Type":"ContainerDied","Data":"f141a22f007be701b06c8a57111c84b0e9571fd657e4e33fa34f3fd7eea613b8"}
Sep 05 02:37:33.851349 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.851046    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f2aa2aac-c629-44c6-9f84-5e577a9659b8","Type":"ContainerDied","Data":"62f9f464cce3319883648dea7e4dbec7e53ff32c693301cbe02c8bb094dcc9b6"}
Sep 05 02:37:33.851349 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.851079    2557 scope.go:117] "RemoveContainer" containerID="f141a22f007be701b06c8a57111c84b0e9571fd657e4e33fa34f3fd7eea613b8"
Sep 05 02:37:33.860035 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.860010    2557 scope.go:117] "RemoveContainer" containerID="e399ebe1989dd6442bca1d0345179b702481a2c27f3151a63e984c66e5ef92db"
Sep 05 02:37:33.868791 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.868771    2557 scope.go:117] "RemoveContainer" containerID="f141a22f007be701b06c8a57111c84b0e9571fd657e4e33fa34f3fd7eea613b8"
Sep 05 02:37:33.869069 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:37:33.869050    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f141a22f007be701b06c8a57111c84b0e9571fd657e4e33fa34f3fd7eea613b8\": container with ID starting with f141a22f007be701b06c8a57111c84b0e9571fd657e4e33fa34f3fd7eea613b8 not found: ID does not exist" containerID="f141a22f007be701b06c8a57111c84b0e9571fd657e4e33fa34f3fd7eea613b8"
Sep 05 02:37:33.869119 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.869078    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f141a22f007be701b06c8a57111c84b0e9571fd657e4e33fa34f3fd7eea613b8"} err="failed to get container status \"f141a22f007be701b06c8a57111c84b0e9571fd657e4e33fa34f3fd7eea613b8\": rpc error: code = NotFound desc = could not find container \"f141a22f007be701b06c8a57111c84b0e9571fd657e4e33fa34f3fd7eea613b8\": container with ID starting with f141a22f007be701b06c8a57111c84b0e9571fd657e4e33fa34f3fd7eea613b8 not found: ID does not exist"
Sep 05 02:37:33.869119 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.869094    2557 scope.go:117] "RemoveContainer" containerID="e399ebe1989dd6442bca1d0345179b702481a2c27f3151a63e984c66e5ef92db"
Sep 05 02:37:33.869343 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:37:33.869318    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e399ebe1989dd6442bca1d0345179b702481a2c27f3151a63e984c66e5ef92db\": container with ID starting with e399ebe1989dd6442bca1d0345179b702481a2c27f3151a63e984c66e5ef92db not found: ID does not exist" containerID="e399ebe1989dd6442bca1d0345179b702481a2c27f3151a63e984c66e5ef92db"
Sep 05 02:37:33.869399 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.869354    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e399ebe1989dd6442bca1d0345179b702481a2c27f3151a63e984c66e5ef92db"} err="failed to get container status \"e399ebe1989dd6442bca1d0345179b702481a2c27f3151a63e984c66e5ef92db\": rpc error: code = NotFound desc = could not find container \"e399ebe1989dd6442bca1d0345179b702481a2c27f3151a63e984c66e5ef92db\": container with ID starting with e399ebe1989dd6442bca1d0345179b702481a2c27f3151a63e984c66e5ef92db not found: ID does not exist"
Sep 05 02:37:33.877546 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.877520    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 05 02:37:33.880955 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:33.880934    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 05 02:37:34.991737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:34.991703    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" path="/var/lib/kubelet/pods/f2aa2aac-c629-44c6-9f84-5e577a9659b8/volumes"
Sep 05 02:37:37.309134 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:37.309089    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 05 02:37:46.925577 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:46.925484    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:37:47.308972 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:47.308862    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 05 02:37:57.309463 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:37:57.309390    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 05 02:38:05.894591 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:05.894557    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:06.064357 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:06.064315    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:07.309802 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:07.309755    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 05 02:38:12.238952 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:12.238916    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:16.945270 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:16.945237    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:38:16.949203 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:16.949184    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:38:17.028499 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:17.028467    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:17.128618 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:17.128584    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:17.309242 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:17.309139    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 05 02:38:17.947307 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:17.947269    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:18.015187 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:18.015136    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:27.309717 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:27.309674    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 05 02:38:29.599825 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.599789    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:29.769517 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.769468    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"]
Sep 05 02:38:29.770058 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.770033    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="main"
Sep 05 02:38:29.770058 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.770058    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="main"
Sep 05 02:38:29.770244 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.770081    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="storage-initializer"
Sep 05 02:38:29.770244 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.770090    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="storage-initializer"
Sep 05 02:38:29.770244 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.770199    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2aa2aac-c629-44c6-9f84-5e577a9659b8" containerName="main"
Sep 05 02:38:29.774978 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.774958    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"
Sep 05 02:38:29.777335 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.777315    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-mgqcr\""
Sep 05 02:38:29.778265 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.778247    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-dq9g7\""
Sep 05 02:38:29.778412 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.778269    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 05 02:38:29.778412 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.778271    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 05 02:38:29.778610 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.778358    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 05 02:38:29.783845 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.783822    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"]
Sep 05 02:38:29.787074 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.787045    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefd9496-7a67-4157-b463-32a07c30f6b8-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq\" (UID: \"fefd9496-7a67-4157-b463-32a07c30f6b8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"
Sep 05 02:38:29.787170 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.787093    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lq8k\" (UniqueName: \"kubernetes.io/projected/fefd9496-7a67-4157-b463-32a07c30f6b8-kube-api-access-4lq8k\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq\" (UID: \"fefd9496-7a67-4157-b463-32a07c30f6b8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"
Sep 05 02:38:29.801410 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.801380    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:29.888523 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.888415    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefd9496-7a67-4157-b463-32a07c30f6b8-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq\" (UID: \"fefd9496-7a67-4157-b463-32a07c30f6b8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"
Sep 05 02:38:29.888523 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.888516    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lq8k\" (UniqueName: \"kubernetes.io/projected/fefd9496-7a67-4157-b463-32a07c30f6b8-kube-api-access-4lq8k\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq\" (UID: \"fefd9496-7a67-4157-b463-32a07c30f6b8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"
Sep 05 02:38:29.891261 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.891226    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefd9496-7a67-4157-b463-32a07c30f6b8-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq\" (UID: \"fefd9496-7a67-4157-b463-32a07c30f6b8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"
Sep 05 02:38:29.896844 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:29.896815    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lq8k\" (UniqueName: \"kubernetes.io/projected/fefd9496-7a67-4157-b463-32a07c30f6b8-kube-api-access-4lq8k\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq\" (UID: \"fefd9496-7a67-4157-b463-32a07c30f6b8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"
Sep 05 02:38:30.086190 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:30.086157    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"
Sep 05 02:38:30.372412 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:30.372382    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"]
Sep 05 02:38:30.374771 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:38:30.374742    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefd9496_7a67_4157_b463_32a07c30f6b8.slice/crio-90c20413d25dbbe57dd7df906cdf6071a16c6240c417b8f2a792154f8d5da602 WatchSource:0}: Error finding container 90c20413d25dbbe57dd7df906cdf6071a16c6240c417b8f2a792154f8d5da602: Status 404 returned error can't find the container with id 90c20413d25dbbe57dd7df906cdf6071a16c6240c417b8f2a792154f8d5da602
Sep 05 02:38:31.187346 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:31.187309    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq" event={"ID":"fefd9496-7a67-4157-b463-32a07c30f6b8","Type":"ContainerStarted","Data":"ba660866ce03701a44f8c1cc725128b25540c006185cd1c4f42e64e09ca796d6"}
Sep 05 02:38:31.187852 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:31.187365    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq" event={"ID":"fefd9496-7a67-4157-b463-32a07c30f6b8","Type":"ContainerStarted","Data":"90c20413d25dbbe57dd7df906cdf6071a16c6240c417b8f2a792154f8d5da602"}
Sep 05 02:38:31.187852 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:31.187406    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"
Sep 05 02:38:31.210126 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:31.210079    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq" podStartSLOduration=2.2100655590000002 podStartE2EDuration="2.210065559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:38:31.209501351 +0000 UTC m=+1214.764657660" watchObservedRunningTime="2026-09-05 02:38:31.210065559 +0000 UTC m=+1214.765221868"
Sep 05 02:38:37.309643 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:37.309593    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 05 02:38:44.657963 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:44.657913    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:47.319728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:47.319693    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:38:47.327737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:47.327711    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:38:52.171917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:52.171879    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"]
Sep 05 02:38:52.181941 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:52.181896    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:38:52.246051 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:38:52.246013    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:39:02.194590 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:02.194558    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"
Sep 05 02:39:03.743141 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:03.743106    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"]
Sep 05 02:39:03.748732 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:03.748695    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:39:03.813451 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:03.813405    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:39:07.196288 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:07.196223    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main" containerID="cri-o://de9b450371fe1c53096b5551795f637b34412b93da4e893a403137fd522b3443" gracePeriod=45
Sep 05 02:39:07.320359 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:07.320323    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 05 02:39:07.952080 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:07.952058    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:39:08.127330 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.127302    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-model-cache\") pod \"8c2eed68-3f0c-4d70-92ca-844a06794fce\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") "
Sep 05 02:39:08.127531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.127341    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-tmp-dir\") pod \"8c2eed68-3f0c-4d70-92ca-844a06794fce\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") "
Sep 05 02:39:08.127531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.127360    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-home\") pod \"8c2eed68-3f0c-4d70-92ca-844a06794fce\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") "
Sep 05 02:39:08.127531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.127379    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-kserve-provision-location\") pod \"8c2eed68-3f0c-4d70-92ca-844a06794fce\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") "
Sep 05 02:39:08.127531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.127409    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lspx\" (UniqueName: \"kubernetes.io/projected/8c2eed68-3f0c-4d70-92ca-844a06794fce-kube-api-access-8lspx\") pod \"8c2eed68-3f0c-4d70-92ca-844a06794fce\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") "
Sep 05 02:39:08.127531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.127484    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2eed68-3f0c-4d70-92ca-844a06794fce-tls-certs\") pod \"8c2eed68-3f0c-4d70-92ca-844a06794fce\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") "
Sep 05 02:39:08.127531 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.127519    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c2eed68-3f0c-4d70-92ca-844a06794fce-cabundle-cert\") pod \"8c2eed68-3f0c-4d70-92ca-844a06794fce\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") "
Sep 05 02:39:08.127853 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.127574    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-dshm\") pod \"8c2eed68-3f0c-4d70-92ca-844a06794fce\" (UID: \"8c2eed68-3f0c-4d70-92ca-844a06794fce\") "
Sep 05 02:39:08.127853 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.127668    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c2eed68-3f0c-4d70-92ca-844a06794fce" (UID: "8c2eed68-3f0c-4d70-92ca-844a06794fce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:39:08.127978 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.127871    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:39:08.128179 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.128144    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2eed68-3f0c-4d70-92ca-844a06794fce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c2eed68-3f0c-4d70-92ca-844a06794fce" (UID: "8c2eed68-3f0c-4d70-92ca-844a06794fce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:39:08.128179 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.128161    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-home" (OuterVolumeSpecName: "home") pod "8c2eed68-3f0c-4d70-92ca-844a06794fce" (UID: "8c2eed68-3f0c-4d70-92ca-844a06794fce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:39:08.130293 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.130265    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2eed68-3f0c-4d70-92ca-844a06794fce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c2eed68-3f0c-4d70-92ca-844a06794fce" (UID: "8c2eed68-3f0c-4d70-92ca-844a06794fce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:39:08.130383 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.130321    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-dshm" (OuterVolumeSpecName: "dshm") pod "8c2eed68-3f0c-4d70-92ca-844a06794fce" (UID: "8c2eed68-3f0c-4d70-92ca-844a06794fce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:39:08.130383 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.130338    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2eed68-3f0c-4d70-92ca-844a06794fce-kube-api-access-8lspx" (OuterVolumeSpecName: "kube-api-access-8lspx") pod "8c2eed68-3f0c-4d70-92ca-844a06794fce" (UID: "8c2eed68-3f0c-4d70-92ca-844a06794fce"). InnerVolumeSpecName "kube-api-access-8lspx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:39:08.141612 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.141582    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c2eed68-3f0c-4d70-92ca-844a06794fce" (UID: "8c2eed68-3f0c-4d70-92ca-844a06794fce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:39:08.152737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.152700    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c2eed68-3f0c-4d70-92ca-844a06794fce" (UID: "8c2eed68-3f0c-4d70-92ca-844a06794fce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:39:08.228705 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.228668    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lspx\" (UniqueName: \"kubernetes.io/projected/8c2eed68-3f0c-4d70-92ca-844a06794fce-kube-api-access-8lspx\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:39:08.228705 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.228699    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2eed68-3f0c-4d70-92ca-844a06794fce-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:39:08.228705 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.228711    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c2eed68-3f0c-4d70-92ca-844a06794fce-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:39:08.229207 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.228719    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:39:08.229207 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.228727    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:39:08.229207 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.228734    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:39:08.229207 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.228741    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c2eed68-3f0c-4d70-92ca-844a06794fce-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:39:08.393905 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.393813    2557 generic.go:356] "Generic (PLEG): container finished" podID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerID="de9b450371fe1c53096b5551795f637b34412b93da4e893a403137fd522b3443" exitCode=0
Sep 05 02:39:08.393905 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.393863    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"
Sep 05 02:39:08.394086 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.393896    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" event={"ID":"8c2eed68-3f0c-4d70-92ca-844a06794fce","Type":"ContainerDied","Data":"de9b450371fe1c53096b5551795f637b34412b93da4e893a403137fd522b3443"}
Sep 05 02:39:08.394086 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.393947    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl" event={"ID":"8c2eed68-3f0c-4d70-92ca-844a06794fce","Type":"ContainerDied","Data":"7335e27d4965733fffff7ad87fa766046a1f4a7a72573239c3ff5ca5c54ed4d5"}
Sep 05 02:39:08.394086 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.393969    2557 scope.go:117] "RemoveContainer" containerID="de9b450371fe1c53096b5551795f637b34412b93da4e893a403137fd522b3443"
Sep 05 02:39:08.403592 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.403573    2557 scope.go:117] "RemoveContainer" containerID="ab17604af58e03b4321709817d8d41fc2e71061de8a59839bdd947e67b75e05a"
Sep 05 02:39:08.411748 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.411731    2557 scope.go:117] "RemoveContainer" containerID="de9b450371fe1c53096b5551795f637b34412b93da4e893a403137fd522b3443"
Sep 05 02:39:08.412018 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:39:08.411992    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de9b450371fe1c53096b5551795f637b34412b93da4e893a403137fd522b3443\": container with ID starting with de9b450371fe1c53096b5551795f637b34412b93da4e893a403137fd522b3443 not found: ID does not exist" containerID="de9b450371fe1c53096b5551795f637b34412b93da4e893a403137fd522b3443"
Sep 05 02:39:08.412111 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.412027    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de9b450371fe1c53096b5551795f637b34412b93da4e893a403137fd522b3443"} err="failed to get container status \"de9b450371fe1c53096b5551795f637b34412b93da4e893a403137fd522b3443\": rpc error: code = NotFound desc = could not find container \"de9b450371fe1c53096b5551795f637b34412b93da4e893a403137fd522b3443\": container with ID starting with de9b450371fe1c53096b5551795f637b34412b93da4e893a403137fd522b3443 not found: ID does not exist"
Sep 05 02:39:08.412111 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.412045    2557 scope.go:117] "RemoveContainer" containerID="ab17604af58e03b4321709817d8d41fc2e71061de8a59839bdd947e67b75e05a"
Sep 05 02:39:08.412323 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:39:08.412304    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab17604af58e03b4321709817d8d41fc2e71061de8a59839bdd947e67b75e05a\": container with ID starting with ab17604af58e03b4321709817d8d41fc2e71061de8a59839bdd947e67b75e05a not found: ID does not exist" containerID="ab17604af58e03b4321709817d8d41fc2e71061de8a59839bdd947e67b75e05a"
Sep 05 02:39:08.412373 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.412328    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab17604af58e03b4321709817d8d41fc2e71061de8a59839bdd947e67b75e05a"} err="failed to get container status \"ab17604af58e03b4321709817d8d41fc2e71061de8a59839bdd947e67b75e05a\": rpc error: code = NotFound desc = could not find container \"ab17604af58e03b4321709817d8d41fc2e71061de8a59839bdd947e67b75e05a\": container with ID starting with ab17604af58e03b4321709817d8d41fc2e71061de8a59839bdd947e67b75e05a not found: ID does not exist"
Sep 05 02:39:08.417467 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.417414    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"]
Sep 05 02:39:08.422385 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.422365    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-fc9648fc9-jdgdl"]
Sep 05 02:39:08.990955 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:08.990879    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" path="/var/lib/kubelet/pods/8c2eed68-3f0c-4d70-92ca-844a06794fce/volumes"
Sep 05 02:39:12.195248 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:12.195205    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq" podUID="fefd9496-7a67-4157-b463-32a07c30f6b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:39:18.764760 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:18.764708    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq" podUID="fefd9496-7a67-4157-b463-32a07c30f6b8" containerName="main" containerID="cri-o://ba660866ce03701a44f8c1cc725128b25540c006185cd1c4f42e64e09ca796d6" gracePeriod=45
Sep 05 02:39:22.195088 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:22.195038    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq" podUID="fefd9496-7a67-4157-b463-32a07c30f6b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:39:32.194818 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:32.194777    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq" podUID="fefd9496-7a67-4157-b463-32a07c30f6b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:39:32.195316 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:32.194879    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"
Sep 05 02:39:42.195175 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:42.195128    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq" podUID="fefd9496-7a67-4157-b463-32a07c30f6b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:39:42.303377 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:42.303341    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:39:42.398893 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:42.398857    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:39:48.206351 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:48.206310    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:39:49.025922 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.025853    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"
Sep 05 02:39:49.163770 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.163733    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lq8k\" (UniqueName: \"kubernetes.io/projected/fefd9496-7a67-4157-b463-32a07c30f6b8-kube-api-access-4lq8k\") pod \"fefd9496-7a67-4157-b463-32a07c30f6b8\" (UID: \"fefd9496-7a67-4157-b463-32a07c30f6b8\") "
Sep 05 02:39:49.163954 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.163854    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefd9496-7a67-4157-b463-32a07c30f6b8-tls-certs\") pod \"fefd9496-7a67-4157-b463-32a07c30f6b8\" (UID: \"fefd9496-7a67-4157-b463-32a07c30f6b8\") "
Sep 05 02:39:49.165931 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.165900    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefd9496-7a67-4157-b463-32a07c30f6b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fefd9496-7a67-4157-b463-32a07c30f6b8" (UID: "fefd9496-7a67-4157-b463-32a07c30f6b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:39:49.166052 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.165927    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefd9496-7a67-4157-b463-32a07c30f6b8-kube-api-access-4lq8k" (OuterVolumeSpecName: "kube-api-access-4lq8k") pod "fefd9496-7a67-4157-b463-32a07c30f6b8" (UID: "fefd9496-7a67-4157-b463-32a07c30f6b8"). InnerVolumeSpecName "kube-api-access-4lq8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:39:49.265372 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.265331    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefd9496-7a67-4157-b463-32a07c30f6b8-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:39:49.265372 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.265369    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lq8k\" (UniqueName: \"kubernetes.io/projected/fefd9496-7a67-4157-b463-32a07c30f6b8-kube-api-access-4lq8k\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:39:49.614941 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.614902    2557 generic.go:356] "Generic (PLEG): container finished" podID="fefd9496-7a67-4157-b463-32a07c30f6b8" containerID="ba660866ce03701a44f8c1cc725128b25540c006185cd1c4f42e64e09ca796d6" exitCode=0
Sep 05 02:39:49.615114 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.614947    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"
Sep 05 02:39:49.615114 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.614982    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq" event={"ID":"fefd9496-7a67-4157-b463-32a07c30f6b8","Type":"ContainerDied","Data":"ba660866ce03701a44f8c1cc725128b25540c006185cd1c4f42e64e09ca796d6"}
Sep 05 02:39:49.615114 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.615042    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq" event={"ID":"fefd9496-7a67-4157-b463-32a07c30f6b8","Type":"ContainerDied","Data":"90c20413d25dbbe57dd7df906cdf6071a16c6240c417b8f2a792154f8d5da602"}
Sep 05 02:39:49.615114 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.615071    2557 scope.go:117] "RemoveContainer" containerID="ba660866ce03701a44f8c1cc725128b25540c006185cd1c4f42e64e09ca796d6"
Sep 05 02:39:49.623940 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.623922    2557 scope.go:117] "RemoveContainer" containerID="ba660866ce03701a44f8c1cc725128b25540c006185cd1c4f42e64e09ca796d6"
Sep 05 02:39:49.624220 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:39:49.624200    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba660866ce03701a44f8c1cc725128b25540c006185cd1c4f42e64e09ca796d6\": container with ID starting with ba660866ce03701a44f8c1cc725128b25540c006185cd1c4f42e64e09ca796d6 not found: ID does not exist" containerID="ba660866ce03701a44f8c1cc725128b25540c006185cd1c4f42e64e09ca796d6"
Sep 05 02:39:49.624274 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.624229    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba660866ce03701a44f8c1cc725128b25540c006185cd1c4f42e64e09ca796d6"} err="failed to get container status \"ba660866ce03701a44f8c1cc725128b25540c006185cd1c4f42e64e09ca796d6\": rpc error: code = NotFound desc = could not find container \"ba660866ce03701a44f8c1cc725128b25540c006185cd1c4f42e64e09ca796d6\": container with ID starting with ba660866ce03701a44f8c1cc725128b25540c006185cd1c4f42e64e09ca796d6 not found: ID does not exist"
Sep 05 02:39:49.636183 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.636160    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"]
Sep 05 02:39:49.639970 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:49.639947    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbvm2fq"]
Sep 05 02:39:50.990741 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:50.990708    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fefd9496-7a67-4157-b463-32a07c30f6b8" path="/var/lib/kubelet/pods/fefd9496-7a67-4157-b463-32a07c30f6b8/volumes"
Sep 05 02:39:53.288989 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:53.288960    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:39:53.386870 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:53.386830    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:39:54.182060 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.182026    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:39:54.282136 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.282098    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:39:54.898995 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.898957    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"]
Sep 05 02:39:54.899489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.899317    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fefd9496-7a67-4157-b463-32a07c30f6b8" containerName="main"
Sep 05 02:39:54.899489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.899330    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fefd9496-7a67-4157-b463-32a07c30f6b8" containerName="main"
Sep 05 02:39:54.899489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.899343    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main"
Sep 05 02:39:54.899489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.899348    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main"
Sep 05 02:39:54.899489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.899358    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="storage-initializer"
Sep 05 02:39:54.899489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.899364    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="storage-initializer"
Sep 05 02:39:54.899489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.899416    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c2eed68-3f0c-4d70-92ca-844a06794fce" containerName="main"
Sep 05 02:39:54.899489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.899422    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fefd9496-7a67-4157-b463-32a07c30f6b8" containerName="main"
Sep 05 02:39:54.910292 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.910265    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:54.912793 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.912752    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:39:54.912955 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.912798    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 05 02:39:54.912955 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.912859    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 05 02:39:54.913079 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.913064    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 05 02:39:54.913276 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.913255    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-rmq52\""
Sep 05 02:39:54.913494 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.913311    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 05 02:39:54.915233 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:54.915210    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"]
Sep 05 02:39:55.000761 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.000727    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:39:55.017854 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.017820    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz9x2\" (UniqueName: \"kubernetes.io/projected/ad9197c6-d267-499f-930a-b1d6e327502a-kube-api-access-wz9x2\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.017977 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.017894    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-dshm\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.017977 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.017937    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9197c6-d267-499f-930a-b1d6e327502a-tls-certs\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.017977 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.017961    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad9197c6-d267-499f-930a-b1d6e327502a-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.018095 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.017982    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-tmp-dir\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.018095 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.018078    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-home\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.018170 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.018126    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-model-cache\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.018170 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.018161    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.119548 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.119507    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-model-cache\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.119729 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.119563    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.119729 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.119681    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz9x2\" (UniqueName: \"kubernetes.io/projected/ad9197c6-d267-499f-930a-b1d6e327502a-kube-api-access-wz9x2\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.119862 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.119735    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-dshm\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.119862 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.119761    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9197c6-d267-499f-930a-b1d6e327502a-tls-certs\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.119862 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.119783    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad9197c6-d267-499f-930a-b1d6e327502a-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.119862 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.119812    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-tmp-dir\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.120079 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.119891    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-home\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.120079 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.119994    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-model-cache\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.120079 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.120069    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.120231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.120215    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-home\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.120270 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.120214    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-tmp-dir\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.120503 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.120483    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad9197c6-d267-499f-930a-b1d6e327502a-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.122079 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.122059    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-dshm\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.122357 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.122338    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9197c6-d267-499f-930a-b1d6e327502a-tls-certs\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.127629 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.127605    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz9x2\" (UniqueName: \"kubernetes.io/projected/ad9197c6-d267-499f-930a-b1d6e327502a-kube-api-access-wz9x2\") pod \"auth-invalid-token-test-kserve-6f9cc9d586-vmxks\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.222316 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.222230    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:39:55.304868 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.304829    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:39:55.523266 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.523235    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"]
Sep 05 02:39:55.525112 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:39:55.525085    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9197c6_d267_499f_930a_b1d6e327502a.slice/crio-f2968d676ea1e7f89e1f7091a76244bf01bf9dfae8c44e6ba83c7de71b35e3a2 WatchSource:0}: Error finding container f2968d676ea1e7f89e1f7091a76244bf01bf9dfae8c44e6ba83c7de71b35e3a2: Status 404 returned error can't find the container with id f2968d676ea1e7f89e1f7091a76244bf01bf9dfae8c44e6ba83c7de71b35e3a2
Sep 05 02:39:55.649759 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.649715    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" event={"ID":"ad9197c6-d267-499f-930a-b1d6e327502a","Type":"ContainerStarted","Data":"cc4f7cbff30236a08a6d9db6b66f23a64124a0bf2bb1cc8750ffa4b4d4dadbf3"}
Sep 05 02:39:55.649933 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:55.649786    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" event={"ID":"ad9197c6-d267-499f-930a-b1d6e327502a","Type":"ContainerStarted","Data":"f2968d676ea1e7f89e1f7091a76244bf01bf9dfae8c44e6ba83c7de71b35e3a2"}
Sep 05 02:39:58.672864 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:58.672830    2557 generic.go:356] "Generic (PLEG): container finished" podID="ad9197c6-d267-499f-930a-b1d6e327502a" containerID="cc4f7cbff30236a08a6d9db6b66f23a64124a0bf2bb1cc8750ffa4b4d4dadbf3" exitCode=0
Sep 05 02:39:58.673251 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:58.672905    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" event={"ID":"ad9197c6-d267-499f-930a-b1d6e327502a","Type":"ContainerDied","Data":"cc4f7cbff30236a08a6d9db6b66f23a64124a0bf2bb1cc8750ffa4b4d4dadbf3"}
Sep 05 02:39:59.668271 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:59.668235    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:39:59.680592 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:59.680557    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" event={"ID":"ad9197c6-d267-499f-930a-b1d6e327502a","Type":"ContainerStarted","Data":"241c730c2cd705f980198b19b8db2e7517d282020cf65a209190e88efbb21072"}
Sep 05 02:39:59.703553 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:39:59.703503    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" podStartSLOduration=3.087303235 podStartE2EDuration="5.703488171s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.616184936s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:39:59.70027547 +0000 UTC m=+1303.255431780" watchObservedRunningTime="2026-09-05 02:39:59.703488171 +0000 UTC m=+1303.258644480"
Sep 05 02:40:00.223204 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:00.223153    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:40:00.493793 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:00.493681    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:05.223214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:05.223181    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:40:05.224893 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:05.224849    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 05 02:40:05.243961 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:05.243928    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:05.302090 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:05.302059    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:05.872868 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:05.872829    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:06.021705 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.021656    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"]
Sep 05 02:40:06.025456 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.025413    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"
Sep 05 02:40:06.029055 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.029025    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-x27zz\""
Sep 05 02:40:06.029196 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.029052    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 05 02:40:06.029196 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.029102    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-9rwcb\""
Sep 05 02:40:06.029196 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.029145    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 05 02:40:06.029416 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.029384    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 05 02:40:06.038513 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.038362    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"]
Sep 05 02:40:06.047828 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.047791    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:06.124992 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.124901    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2wgj\" (UniqueName: \"kubernetes.io/projected/2d43fe7c-f0c7-4ffa-9f73-faba49deb320-kube-api-access-d2wgj\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7\" (UID: \"2d43fe7c-f0c7-4ffa-9f73-faba49deb320\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"
Sep 05 02:40:06.124992 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.124971    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d43fe7c-f0c7-4ffa-9f73-faba49deb320-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7\" (UID: \"2d43fe7c-f0c7-4ffa-9f73-faba49deb320\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"
Sep 05 02:40:06.225874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.225828    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2wgj\" (UniqueName: \"kubernetes.io/projected/2d43fe7c-f0c7-4ffa-9f73-faba49deb320-kube-api-access-d2wgj\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7\" (UID: \"2d43fe7c-f0c7-4ffa-9f73-faba49deb320\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"
Sep 05 02:40:06.226326 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.225929    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d43fe7c-f0c7-4ffa-9f73-faba49deb320-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7\" (UID: \"2d43fe7c-f0c7-4ffa-9f73-faba49deb320\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"
Sep 05 02:40:06.228604 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.228572    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d43fe7c-f0c7-4ffa-9f73-faba49deb320-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7\" (UID: \"2d43fe7c-f0c7-4ffa-9f73-faba49deb320\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"
Sep 05 02:40:06.233995 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.233961    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2wgj\" (UniqueName: \"kubernetes.io/projected/2d43fe7c-f0c7-4ffa-9f73-faba49deb320-kube-api-access-d2wgj\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7\" (UID: \"2d43fe7c-f0c7-4ffa-9f73-faba49deb320\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"
Sep 05 02:40:06.344328 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.344293    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"
Sep 05 02:40:06.621501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.621395    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"]
Sep 05 02:40:06.622589 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:40:06.622563    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d43fe7c_f0c7_4ffa_9f73_faba49deb320.slice/crio-918eb9327767795e6b29f090e0a471e75f715acac9f61419b9f67255361b442f WatchSource:0}: Error finding container 918eb9327767795e6b29f090e0a471e75f715acac9f61419b9f67255361b442f: Status 404 returned error can't find the container with id 918eb9327767795e6b29f090e0a471e75f715acac9f61419b9f67255361b442f
Sep 05 02:40:06.721619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.721580    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7" event={"ID":"2d43fe7c-f0c7-4ffa-9f73-faba49deb320","Type":"ContainerStarted","Data":"e0c341193339b830213dc7490b44ceebba7b93b9930221e5f9e18f9be74101fe"}
Sep 05 02:40:06.721815 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.721648    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7" event={"ID":"2d43fe7c-f0c7-4ffa-9f73-faba49deb320","Type":"ContainerStarted","Data":"918eb9327767795e6b29f090e0a471e75f715acac9f61419b9f67255361b442f"}
Sep 05 02:40:06.721815 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.721688    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"
Sep 05 02:40:06.742367 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:06.742304    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7" podStartSLOduration=1.742282999 podStartE2EDuration="1.742282999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:40:06.738425328 +0000 UTC m=+1310.293581637" watchObservedRunningTime="2026-09-05 02:40:06.742282999 +0000 UTC m=+1310.297439310"
Sep 05 02:40:15.223279 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:15.223226    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 05 02:40:15.900811 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:15.900765    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:25.222775 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:25.222709    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 05 02:40:26.908795 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:26.908761    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:35.223329 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:35.223275    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 05 02:40:37.729184 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:37.729154    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"
Sep 05 02:40:39.332952 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:39.332862    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:39.349429 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:39.349384    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"]
Sep 05 02:40:39.429019 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:39.428969    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:40:45.223695 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:45.223649    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 05 02:40:47.728585 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:47.728538    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7" podUID="2d43fe7c-f0c7-4ffa-9f73-faba49deb320" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:40:54.370919 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:54.370873    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7" podUID="2d43fe7c-f0c7-4ffa-9f73-faba49deb320" containerName="main" containerID="cri-o://e0c341193339b830213dc7490b44ceebba7b93b9930221e5f9e18f9be74101fe" gracePeriod=45
Sep 05 02:40:55.222902 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:55.222864    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 05 02:40:57.729287 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:40:57.729246    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7" podUID="2d43fe7c-f0c7-4ffa-9f73-faba49deb320" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:41:05.223264 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:05.223217    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 05 02:41:07.728693 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:07.728648    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7" podUID="2d43fe7c-f0c7-4ffa-9f73-faba49deb320" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:41:07.729084 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:07.728755    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"
Sep 05 02:41:15.223693 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:15.223645    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 05 02:41:17.728413 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:17.728370    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7" podUID="2d43fe7c-f0c7-4ffa-9f73-faba49deb320" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:41:24.633059 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:24.633000    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"
Sep 05 02:41:24.739870 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:24.739832    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2wgj\" (UniqueName: \"kubernetes.io/projected/2d43fe7c-f0c7-4ffa-9f73-faba49deb320-kube-api-access-d2wgj\") pod \"2d43fe7c-f0c7-4ffa-9f73-faba49deb320\" (UID: \"2d43fe7c-f0c7-4ffa-9f73-faba49deb320\") "
Sep 05 02:41:24.740022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:24.739891    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d43fe7c-f0c7-4ffa-9f73-faba49deb320-tls-certs\") pod \"2d43fe7c-f0c7-4ffa-9f73-faba49deb320\" (UID: \"2d43fe7c-f0c7-4ffa-9f73-faba49deb320\") "
Sep 05 02:41:24.742137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:24.742107    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d43fe7c-f0c7-4ffa-9f73-faba49deb320-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d43fe7c-f0c7-4ffa-9f73-faba49deb320" (UID: "2d43fe7c-f0c7-4ffa-9f73-faba49deb320"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:41:24.742137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:24.742122    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d43fe7c-f0c7-4ffa-9f73-faba49deb320-kube-api-access-d2wgj" (OuterVolumeSpecName: "kube-api-access-d2wgj") pod "2d43fe7c-f0c7-4ffa-9f73-faba49deb320" (UID: "2d43fe7c-f0c7-4ffa-9f73-faba49deb320"). InnerVolumeSpecName "kube-api-access-d2wgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:41:24.840626 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:24.840587    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2wgj\" (UniqueName: \"kubernetes.io/projected/2d43fe7c-f0c7-4ffa-9f73-faba49deb320-kube-api-access-d2wgj\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:41:24.840626 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:24.840627    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d43fe7c-f0c7-4ffa-9f73-faba49deb320-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:41:25.172878 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:25.172832    2557 generic.go:356] "Generic (PLEG): container finished" podID="2d43fe7c-f0c7-4ffa-9f73-faba49deb320" containerID="e0c341193339b830213dc7490b44ceebba7b93b9930221e5f9e18f9be74101fe" exitCode=0
Sep 05 02:41:25.173071 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:25.172899    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7" event={"ID":"2d43fe7c-f0c7-4ffa-9f73-faba49deb320","Type":"ContainerDied","Data":"e0c341193339b830213dc7490b44ceebba7b93b9930221e5f9e18f9be74101fe"}
Sep 05 02:41:25.173071 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:25.172918    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"
Sep 05 02:41:25.173071 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:25.172947    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7" event={"ID":"2d43fe7c-f0c7-4ffa-9f73-faba49deb320","Type":"ContainerDied","Data":"918eb9327767795e6b29f090e0a471e75f715acac9f61419b9f67255361b442f"}
Sep 05 02:41:25.173071 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:25.172971    2557 scope.go:117] "RemoveContainer" containerID="e0c341193339b830213dc7490b44ceebba7b93b9930221e5f9e18f9be74101fe"
Sep 05 02:41:25.182220 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:25.182198    2557 scope.go:117] "RemoveContainer" containerID="e0c341193339b830213dc7490b44ceebba7b93b9930221e5f9e18f9be74101fe"
Sep 05 02:41:25.182550 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:41:25.182521    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c341193339b830213dc7490b44ceebba7b93b9930221e5f9e18f9be74101fe\": container with ID starting with e0c341193339b830213dc7490b44ceebba7b93b9930221e5f9e18f9be74101fe not found: ID does not exist" containerID="e0c341193339b830213dc7490b44ceebba7b93b9930221e5f9e18f9be74101fe"
Sep 05 02:41:25.182622 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:25.182563    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c341193339b830213dc7490b44ceebba7b93b9930221e5f9e18f9be74101fe"} err="failed to get container status \"e0c341193339b830213dc7490b44ceebba7b93b9930221e5f9e18f9be74101fe\": rpc error: code = NotFound desc = could not find container \"e0c341193339b830213dc7490b44ceebba7b93b9930221e5f9e18f9be74101fe\": container with ID starting with e0c341193339b830213dc7490b44ceebba7b93b9930221e5f9e18f9be74101fe not found: ID does not exist"
Sep 05 02:41:25.191009 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:25.190980    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"]
Sep 05 02:41:25.194464 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:25.194419    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schelwss7"]
Sep 05 02:41:25.223211 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:25.223183    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 05 02:41:25.742091 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:25.742053    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:26.990217 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:26.990182    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d43fe7c-f0c7-4ffa-9f73-faba49deb320" path="/var/lib/kubelet/pods/2d43fe7c-f0c7-4ffa-9f73-faba49deb320/volumes"
Sep 05 02:41:29.092151 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:29.092115    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:29.186497 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:29.186457    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:35.223060 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:35.223009    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 05 02:41:35.333157 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:35.333121    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:39.907646 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:39.907608    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"]
Sep 05 02:41:39.908152 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:39.908133    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d43fe7c-f0c7-4ffa-9f73-faba49deb320" containerName="main"
Sep 05 02:41:39.908223 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:39.908153    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d43fe7c-f0c7-4ffa-9f73-faba49deb320" containerName="main"
Sep 05 02:41:39.908285 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:39.908273    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d43fe7c-f0c7-4ffa-9f73-faba49deb320" containerName="main"
Sep 05 02:41:39.912058 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:39.912037    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:39.915314 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:39.915289    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 05 02:41:39.915462 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:39.915293    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 05 02:41:39.915462 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:39.915367    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 05 02:41:39.915462 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:39.915294    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-bzj8h\""
Sep 05 02:41:39.920907 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:39.920877    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"]
Sep 05 02:41:39.999380 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:39.999347    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:40.079819 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.079776    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.080022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.079847    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029a65ec-c725-4d78-b9ea-276619c78bb5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.080022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.079929    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.080022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.079985    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.080222 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.080023    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f6kb\" (UniqueName: \"kubernetes.io/projected/029a65ec-c725-4d78-b9ea-276619c78bb5-kube-api-access-8f6kb\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.080222 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.080046    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.181582 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.181463    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.181582 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.181510    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.181582 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.181534    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f6kb\" (UniqueName: \"kubernetes.io/projected/029a65ec-c725-4d78-b9ea-276619c78bb5-kube-api-access-8f6kb\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.181582 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.181558    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.181960 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.181639    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.181960 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.181729    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029a65ec-c725-4d78-b9ea-276619c78bb5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.182103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.182069    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.182190 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.182166    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.182386 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.182363    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.184041 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.184020    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.184369 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.184348    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029a65ec-c725-4d78-b9ea-276619c78bb5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.189476 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.189456    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f6kb\" (UniqueName: \"kubernetes.io/projected/029a65ec-c725-4d78-b9ea-276619c78bb5-kube-api-access-8f6kb\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.223758 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.223475    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:40.251742 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.251701    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:40.499969 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:40.499940    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"]
Sep 05 02:41:40.503048 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:41:40.503008    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029a65ec_c725_4d78_b9ea_276619c78bb5.slice/crio-d3eb963c813b455c5129d38d527028f1b8a1f728881be340a58c244c0eba2f1f WatchSource:0}: Error finding container d3eb963c813b455c5129d38d527028f1b8a1f728881be340a58c244c0eba2f1f: Status 404 returned error can't find the container with id d3eb963c813b455c5129d38d527028f1b8a1f728881be340a58c244c0eba2f1f
Sep 05 02:41:41.224911 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:41.224875    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:41.265007 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:41.264969    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j" event={"ID":"029a65ec-c725-4d78-b9ea-276619c78bb5","Type":"ContainerStarted","Data":"55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5"}
Sep 05 02:41:41.265200 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:41.265040    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j" event={"ID":"029a65ec-c725-4d78-b9ea-276619c78bb5","Type":"ContainerStarted","Data":"d3eb963c813b455c5129d38d527028f1b8a1f728881be340a58c244c0eba2f1f"}
Sep 05 02:41:41.289776 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:41.289698    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j" podStartSLOduration=2.289682446 podStartE2EDuration="2.289682446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:41:41.286068336 +0000 UTC m=+1404.841224657" watchObservedRunningTime="2026-09-05 02:41:41.289682446 +0000 UTC m=+1404.844838798"
Sep 05 02:41:41.353578 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:41.353532    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:42.224388 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:42.224349    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:45.223725 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:45.223679    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 05 02:41:47.629033 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:47.628997    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:50.223929 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:50.223887    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:50.236620 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:50.236591    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:50.331062 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:50.331027    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:41:55.233596 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:55.233560    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:41:55.241403 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:55.241373    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:41:57.483691 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:57.483655    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"]
Sep 05 02:41:57.484172 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:57.483805    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:41:57.618110 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:41:57.618070    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:12.507350 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:12.507224    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main" containerID="cri-o://241c730c2cd705f980198b19b8db2e7517d282020cf65a209190e88efbb21072" gracePeriod=45
Sep 05 02:42:13.270465 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.270420    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:42:13.403517 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.403481    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad9197c6-d267-499f-930a-b1d6e327502a-cabundle-cert\") pod \"ad9197c6-d267-499f-930a-b1d6e327502a\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") "
Sep 05 02:42:13.403717 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.403537    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz9x2\" (UniqueName: \"kubernetes.io/projected/ad9197c6-d267-499f-930a-b1d6e327502a-kube-api-access-wz9x2\") pod \"ad9197c6-d267-499f-930a-b1d6e327502a\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") "
Sep 05 02:42:13.403717 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.403573    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9197c6-d267-499f-930a-b1d6e327502a-tls-certs\") pod \"ad9197c6-d267-499f-930a-b1d6e327502a\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") "
Sep 05 02:42:13.403717 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.403596    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-tmp-dir\") pod \"ad9197c6-d267-499f-930a-b1d6e327502a\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") "
Sep 05 02:42:13.403717 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.403614    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-home\") pod \"ad9197c6-d267-499f-930a-b1d6e327502a\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") "
Sep 05 02:42:13.403717 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.403649    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-dshm\") pod \"ad9197c6-d267-499f-930a-b1d6e327502a\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") "
Sep 05 02:42:13.403717 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.403672    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-kserve-provision-location\") pod \"ad9197c6-d267-499f-930a-b1d6e327502a\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") "
Sep 05 02:42:13.404051 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.403725    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-model-cache\") pod \"ad9197c6-d267-499f-930a-b1d6e327502a\" (UID: \"ad9197c6-d267-499f-930a-b1d6e327502a\") "
Sep 05 02:42:13.404188 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.404158    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9197c6-d267-499f-930a-b1d6e327502a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad9197c6-d267-499f-930a-b1d6e327502a" (UID: "ad9197c6-d267-499f-930a-b1d6e327502a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:42:13.404510 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.404409    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-home" (OuterVolumeSpecName: "home") pod "ad9197c6-d267-499f-930a-b1d6e327502a" (UID: "ad9197c6-d267-499f-930a-b1d6e327502a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:42:13.404634 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.404511    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad9197c6-d267-499f-930a-b1d6e327502a" (UID: "ad9197c6-d267-499f-930a-b1d6e327502a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:42:13.404778 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.404759    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad9197c6-d267-499f-930a-b1d6e327502a-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:42:13.404863 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.404785    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:42:13.404863 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.404799    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:42:13.406651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.406623    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-dshm" (OuterVolumeSpecName: "dshm") pod "ad9197c6-d267-499f-930a-b1d6e327502a" (UID: "ad9197c6-d267-499f-930a-b1d6e327502a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:42:13.406773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.406641    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9197c6-d267-499f-930a-b1d6e327502a-kube-api-access-wz9x2" (OuterVolumeSpecName: "kube-api-access-wz9x2") pod "ad9197c6-d267-499f-930a-b1d6e327502a" (UID: "ad9197c6-d267-499f-930a-b1d6e327502a"). InnerVolumeSpecName "kube-api-access-wz9x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:42:13.406773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.406705    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad9197c6-d267-499f-930a-b1d6e327502a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad9197c6-d267-499f-930a-b1d6e327502a" (UID: "ad9197c6-d267-499f-930a-b1d6e327502a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:42:13.417449 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.417405    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad9197c6-d267-499f-930a-b1d6e327502a" (UID: "ad9197c6-d267-499f-930a-b1d6e327502a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:42:13.428262 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.428219    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad9197c6-d267-499f-930a-b1d6e327502a" (UID: "ad9197c6-d267-499f-930a-b1d6e327502a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:42:13.453365 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.453328    2557 generic.go:356] "Generic (PLEG): container finished" podID="ad9197c6-d267-499f-930a-b1d6e327502a" containerID="241c730c2cd705f980198b19b8db2e7517d282020cf65a209190e88efbb21072" exitCode=0
Sep 05 02:42:13.453583 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.453379    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" event={"ID":"ad9197c6-d267-499f-930a-b1d6e327502a","Type":"ContainerDied","Data":"241c730c2cd705f980198b19b8db2e7517d282020cf65a209190e88efbb21072"}
Sep 05 02:42:13.453583 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.453395    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"
Sep 05 02:42:13.453583 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.453483    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" event={"ID":"ad9197c6-d267-499f-930a-b1d6e327502a","Type":"ContainerDied","Data":"f2968d676ea1e7f89e1f7091a76244bf01bf9dfae8c44e6ba83c7de71b35e3a2"}
Sep 05 02:42:13.453583 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.453517    2557 scope.go:117] "RemoveContainer" containerID="241c730c2cd705f980198b19b8db2e7517d282020cf65a209190e88efbb21072"
Sep 05 02:42:13.464446 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.464414    2557 scope.go:117] "RemoveContainer" containerID="cc4f7cbff30236a08a6d9db6b66f23a64124a0bf2bb1cc8750ffa4b4d4dadbf3"
Sep 05 02:42:13.474226 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.474200    2557 scope.go:117] "RemoveContainer" containerID="241c730c2cd705f980198b19b8db2e7517d282020cf65a209190e88efbb21072"
Sep 05 02:42:13.474671 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:42:13.474642    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"241c730c2cd705f980198b19b8db2e7517d282020cf65a209190e88efbb21072\": container with ID starting with 241c730c2cd705f980198b19b8db2e7517d282020cf65a209190e88efbb21072 not found: ID does not exist" containerID="241c730c2cd705f980198b19b8db2e7517d282020cf65a209190e88efbb21072"
Sep 05 02:42:13.474772 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.474685    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241c730c2cd705f980198b19b8db2e7517d282020cf65a209190e88efbb21072"} err="failed to get container status \"241c730c2cd705f980198b19b8db2e7517d282020cf65a209190e88efbb21072\": rpc error: code = NotFound desc = could not find container \"241c730c2cd705f980198b19b8db2e7517d282020cf65a209190e88efbb21072\": container with ID starting with 241c730c2cd705f980198b19b8db2e7517d282020cf65a209190e88efbb21072 not found: ID does not exist"
Sep 05 02:42:13.474772 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.474711    2557 scope.go:117] "RemoveContainer" containerID="cc4f7cbff30236a08a6d9db6b66f23a64124a0bf2bb1cc8750ffa4b4d4dadbf3"
Sep 05 02:42:13.475087 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:42:13.475061    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4f7cbff30236a08a6d9db6b66f23a64124a0bf2bb1cc8750ffa4b4d4dadbf3\": container with ID starting with cc4f7cbff30236a08a6d9db6b66f23a64124a0bf2bb1cc8750ffa4b4d4dadbf3 not found: ID does not exist" containerID="cc4f7cbff30236a08a6d9db6b66f23a64124a0bf2bb1cc8750ffa4b4d4dadbf3"
Sep 05 02:42:13.475146 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.475095    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4f7cbff30236a08a6d9db6b66f23a64124a0bf2bb1cc8750ffa4b4d4dadbf3"} err="failed to get container status \"cc4f7cbff30236a08a6d9db6b66f23a64124a0bf2bb1cc8750ffa4b4d4dadbf3\": rpc error: code = NotFound desc = could not find container \"cc4f7cbff30236a08a6d9db6b66f23a64124a0bf2bb1cc8750ffa4b4d4dadbf3\": container with ID starting with cc4f7cbff30236a08a6d9db6b66f23a64124a0bf2bb1cc8750ffa4b4d4dadbf3 not found: ID does not exist"
Sep 05 02:42:13.476692 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.476664    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"]
Sep 05 02:42:13.484167 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.484141    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks"]
Sep 05 02:42:13.505555 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.505522    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz9x2\" (UniqueName: \"kubernetes.io/projected/ad9197c6-d267-499f-930a-b1d6e327502a-kube-api-access-wz9x2\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:42:13.505555 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.505555    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9197c6-d267-499f-930a-b1d6e327502a-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:42:13.505766 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.505571    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:42:13.505766 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.505583    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:42:13.505766 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.505597    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad9197c6-d267-499f-930a-b1d6e327502a-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:42:13.554338 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.554293    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"]
Sep 05 02:42:13.557389 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.557350    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:13.565226 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:42:13.565191    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:42:13.565226 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:42:13.565226 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:42:13.565226 ip-10-0-143-128 kubenswrapper[2557]:  > containerID="55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5" cmd=["/bin/sleep","15"]
Sep 05 02:42:13.565538 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.565247    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j" message="" err=<
Sep 05 02:42:13.565538 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:42:13.565538 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:42:13.565538 ip-10-0-143-128 kubenswrapper[2557]:  >
Sep 05 02:42:13.565538 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:42:13.565265    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:42:13.565538 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:42:13.565538 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:42:13.565538 ip-10-0-143-128 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j" podUID="029a65ec-c725-4d78-b9ea-276619c78bb5" containerName="main" containerID="cri-o://55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5"
Sep 05 02:42:13.565538 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.565315    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j" podUID="029a65ec-c725-4d78-b9ea-276619c78bb5" containerName="main" containerID="cri-o://55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5" gracePeriod=60
Sep 05 02:42:13.657685 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.657576    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:13.908085 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:13.908021    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:42:14.010195 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.010149    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f6kb\" (UniqueName: \"kubernetes.io/projected/029a65ec-c725-4d78-b9ea-276619c78bb5-kube-api-access-8f6kb\") pod \"029a65ec-c725-4d78-b9ea-276619c78bb5\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") "
Sep 05 02:42:14.010418 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.010223    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-dshm\") pod \"029a65ec-c725-4d78-b9ea-276619c78bb5\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") "
Sep 05 02:42:14.010418 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.010268    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029a65ec-c725-4d78-b9ea-276619c78bb5-tls-certs\") pod \"029a65ec-c725-4d78-b9ea-276619c78bb5\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") "
Sep 05 02:42:14.010418 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.010293    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-tmp-dir\") pod \"029a65ec-c725-4d78-b9ea-276619c78bb5\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") "
Sep 05 02:42:14.010418 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.010321    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-model-cache\") pod \"029a65ec-c725-4d78-b9ea-276619c78bb5\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") "
Sep 05 02:42:14.010418 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.010343    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-home\") pod \"029a65ec-c725-4d78-b9ea-276619c78bb5\" (UID: \"029a65ec-c725-4d78-b9ea-276619c78bb5\") "
Sep 05 02:42:14.010719 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.010656    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-model-cache" (OuterVolumeSpecName: "model-cache") pod "029a65ec-c725-4d78-b9ea-276619c78bb5" (UID: "029a65ec-c725-4d78-b9ea-276619c78bb5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:42:14.010719 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.010681    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-home" (OuterVolumeSpecName: "home") pod "029a65ec-c725-4d78-b9ea-276619c78bb5" (UID: "029a65ec-c725-4d78-b9ea-276619c78bb5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:42:14.010719 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.010664    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "029a65ec-c725-4d78-b9ea-276619c78bb5" (UID: "029a65ec-c725-4d78-b9ea-276619c78bb5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:42:14.010844 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.010752    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:42:14.010844 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.010774    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:42:14.010844 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.010782    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:42:14.012640 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.012610    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-dshm" (OuterVolumeSpecName: "dshm") pod "029a65ec-c725-4d78-b9ea-276619c78bb5" (UID: "029a65ec-c725-4d78-b9ea-276619c78bb5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:42:14.012717 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.012699    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029a65ec-c725-4d78-b9ea-276619c78bb5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "029a65ec-c725-4d78-b9ea-276619c78bb5" (UID: "029a65ec-c725-4d78-b9ea-276619c78bb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:42:14.012760 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.012737    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029a65ec-c725-4d78-b9ea-276619c78bb5-kube-api-access-8f6kb" (OuterVolumeSpecName: "kube-api-access-8f6kb") pod "029a65ec-c725-4d78-b9ea-276619c78bb5" (UID: "029a65ec-c725-4d78-b9ea-276619c78bb5"). InnerVolumeSpecName "kube-api-access-8f6kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:42:14.111334 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.111296    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f6kb\" (UniqueName: \"kubernetes.io/projected/029a65ec-c725-4d78-b9ea-276619c78bb5-kube-api-access-8f6kb\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:42:14.111334 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.111329    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029a65ec-c725-4d78-b9ea-276619c78bb5-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:42:14.111334 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.111341    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029a65ec-c725-4d78-b9ea-276619c78bb5-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:42:14.235180 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.235088    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6f9cc9d586-vmxks" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 05 02:42:14.462762 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.462722    2557 generic.go:356] "Generic (PLEG): container finished" podID="029a65ec-c725-4d78-b9ea-276619c78bb5" containerID="55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5" exitCode=0
Sep 05 02:42:14.462961 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.462862    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j" event={"ID":"029a65ec-c725-4d78-b9ea-276619c78bb5","Type":"ContainerDied","Data":"55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5"}
Sep 05 02:42:14.462961 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.462950    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j" event={"ID":"029a65ec-c725-4d78-b9ea-276619c78bb5","Type":"ContainerDied","Data":"d3eb963c813b455c5129d38d527028f1b8a1f728881be340a58c244c0eba2f1f"}
Sep 05 02:42:14.463077 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.462989    2557 scope.go:117] "RemoveContainer" containerID="55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5"
Sep 05 02:42:14.463355 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.463183    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"
Sep 05 02:42:14.477228 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.477208    2557 scope.go:117] "RemoveContainer" containerID="55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5"
Sep 05 02:42:14.477525 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:42:14.477502    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5\": container with ID starting with 55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5 not found: ID does not exist" containerID="55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5"
Sep 05 02:42:14.477597 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.477533    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5"} err="failed to get container status \"55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5\": rpc error: code = NotFound desc = could not find container \"55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5\": container with ID starting with 55ab1907cecc0b0d5404c9bd0116bc3251c8de3d62f36d368808eee8530651a5 not found: ID does not exist"
Sep 05 02:42:14.489716 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.489632    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"]
Sep 05 02:42:14.495120 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.495093    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-5xv7j"]
Sep 05 02:42:14.990321 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.990274    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="029a65ec-c725-4d78-b9ea-276619c78bb5" path="/var/lib/kubelet/pods/029a65ec-c725-4d78-b9ea-276619c78bb5/volumes"
Sep 05 02:42:14.990913 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:14.990889    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" path="/var/lib/kubelet/pods/ad9197c6-d267-499f-930a-b1d6e327502a/volumes"
Sep 05 02:42:47.404994 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:47.404951    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:47.503926 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:47.503884    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:53.661987 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:53.661952    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:59.247906 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:59.247868    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:42:59.318675 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:42:59.318636    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:00.761033 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.760993    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"]
Sep 05 02:43:00.761407 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.761340    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main"
Sep 05 02:43:00.761407 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.761352    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main"
Sep 05 02:43:00.761407 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.761378    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="storage-initializer"
Sep 05 02:43:00.761407 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.761384    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="storage-initializer"
Sep 05 02:43:00.761407 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.761390    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029a65ec-c725-4d78-b9ea-276619c78bb5" containerName="main"
Sep 05 02:43:00.761407 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.761396    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="029a65ec-c725-4d78-b9ea-276619c78bb5" containerName="main"
Sep 05 02:43:00.761632 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.761472    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad9197c6-d267-499f-930a-b1d6e327502a" containerName="main"
Sep 05 02:43:00.761632 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.761482    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="029a65ec-c725-4d78-b9ea-276619c78bb5" containerName="main"
Sep 05 02:43:00.764596 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.764578    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:00.767606 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.767579    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-rgwqs\""
Sep 05 02:43:00.767755 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.767643    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:43:00.767755 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.767655    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 05 02:43:00.768329 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.768307    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 05 02:43:00.768465 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.768327    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 05 02:43:00.768465 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.768349    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 05 02:43:00.786582 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.786554    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"]
Sep 05 02:43:00.832092 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.832060    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:00.914992 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.914956    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-dshm\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:00.914992 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.914996    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-tls-certs\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:00.915244 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.915020    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-tmp-dir\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:00.915244 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.915101    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-home\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:00.915244 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.915167    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-kserve-provision-location\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:00.915244 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.915196    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-model-cache\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:00.915244 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.915225    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv5dj\" (UniqueName: \"kubernetes.io/projected/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-kube-api-access-sv5dj\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:00.915429 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:00.915277    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-cabundle-cert\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.016218 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.016137    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-model-cache\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.016218 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.016174    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv5dj\" (UniqueName: \"kubernetes.io/projected/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-kube-api-access-sv5dj\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.016218 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.016204    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-cabundle-cert\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.016516 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.016297    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-dshm\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.016516 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.016324    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-tls-certs\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.016516 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.016354    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-tmp-dir\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.016516 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.016382    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-home\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.016773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.016577    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-kserve-provision-location\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.016773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.016709    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-tmp-dir\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.016773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.016765    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-home\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.016944 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.016898    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-model-cache\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.017002 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.016967    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-kserve-provision-location\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.017081 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.017058    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-cabundle-cert\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.019269 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.019244    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-dshm\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.019557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.019250    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-tls-certs\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.024736 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.024703    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv5dj\" (UniqueName: \"kubernetes.io/projected/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-kube-api-access-sv5dj\") pod \"auth-disabled-test-kserve-86cf5c7489-27xnm\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.046071 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.046042    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:01.076935 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.076910    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:01.355644 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.355614    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"]
Sep 05 02:43:01.356911 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:43:01.356887    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4649cd_5bda_470e_b2f0_8a29739dc8f5.slice/crio-18c16af2095cae5278bcac2181b8d9bd5137666a565b637f865a629f5fbe3b61 WatchSource:0}: Error finding container 18c16af2095cae5278bcac2181b8d9bd5137666a565b637f865a629f5fbe3b61: Status 404 returned error can't find the container with id 18c16af2095cae5278bcac2181b8d9bd5137666a565b637f865a629f5fbe3b61
Sep 05 02:43:01.705739 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.705644    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" event={"ID":"6c4649cd-5bda-470e-b2f0-8a29739dc8f5","Type":"ContainerStarted","Data":"fcc35ada4d7a6786873d1959ff230d69194040447dcaa16edff821ee81937a3b"}
Sep 05 02:43:01.705739 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:01.705702    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" event={"ID":"6c4649cd-5bda-470e-b2f0-8a29739dc8f5","Type":"ContainerStarted","Data":"18c16af2095cae5278bcac2181b8d9bd5137666a565b637f865a629f5fbe3b61"}
Sep 05 02:43:03.138755 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:03.138710    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:03.290184 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:03.290149    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:04.725291 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:04.725256    2557 generic.go:356] "Generic (PLEG): container finished" podID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerID="fcc35ada4d7a6786873d1959ff230d69194040447dcaa16edff821ee81937a3b" exitCode=0
Sep 05 02:43:04.725722 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:04.725328    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" event={"ID":"6c4649cd-5bda-470e-b2f0-8a29739dc8f5","Type":"ContainerDied","Data":"fcc35ada4d7a6786873d1959ff230d69194040447dcaa16edff821ee81937a3b"}
Sep 05 02:43:05.732951 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:05.732916    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" event={"ID":"6c4649cd-5bda-470e-b2f0-8a29739dc8f5","Type":"ContainerStarted","Data":"3e9dfd8944af1ca22ef91d1ed3c9f718d5be668e46e47fe985b0f27f6b0b8276"}
Sep 05 02:43:05.756369 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:05.756289    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" podStartSLOduration=3.080947056 podStartE2EDuration="5.756256415s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.675309359s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:43:05.754643604 +0000 UTC m=+1489.309799936" watchObservedRunningTime="2026-09-05 02:43:05.756256415 +0000 UTC m=+1489.311412724"
Sep 05 02:43:06.077032 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:06.076993    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:09.046413 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:09.046366    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:10.028365 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:10.028324    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:11.077182 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:11.077144    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:43:11.078975 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:11.078929    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 05 02:43:14.862836 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.862795    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:14.914401 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.914365    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"]
Sep 05 02:43:14.923731 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.923703    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:14.927708 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.927682    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 05 02:43:14.928289 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.928264    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 05 02:43:14.928397 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.928356    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 05 02:43:14.928488 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.928419    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-99qr6\""
Sep 05 02:43:14.935288 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.935262    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"]
Sep 05 02:43:14.938022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.938002    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:14.938133 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.938028    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b55ab26-0b14-416d-8aa0-e528fca11a6a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:14.938133 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.938045    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2rv6\" (UniqueName: \"kubernetes.io/projected/9b55ab26-0b14-416d-8aa0-e528fca11a6a-kube-api-access-q2rv6\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:14.938248 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.938150    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:14.938248 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.938181    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:14.938248 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.938196    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:14.973046 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:14.973015    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:15.038782 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.038752    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:15.038782 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.038785    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b55ab26-0b14-416d-8aa0-e528fca11a6a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:15.039026 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.038804    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2rv6\" (UniqueName: \"kubernetes.io/projected/9b55ab26-0b14-416d-8aa0-e528fca11a6a-kube-api-access-q2rv6\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:15.039026 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.038843    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:15.039026 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.038876    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:15.039026 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.038896    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:15.039255 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.039228    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:15.039328 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.039304    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:15.039328 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.039317    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:15.041133 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.041113    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:15.041564 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.041543    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b55ab26-0b14-416d-8aa0-e528fca11a6a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:15.047346 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.047322    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2rv6\" (UniqueName: \"kubernetes.io/projected/9b55ab26-0b14-416d-8aa0-e528fca11a6a-kube-api-access-q2rv6\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-bjd47\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:15.190821 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.190738    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"]
Sep 05 02:43:15.193718 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.193697    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"
Sep 05 02:43:15.197050 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.197030    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-xm9jv\""
Sep 05 02:43:15.234657 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.234631    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:15.241737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.241708    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czqpr\" (UniqueName: \"kubernetes.io/projected/f9d773b9-3cb1-4cba-afac-ebb1e089fac8-kube-api-access-czqpr\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w\" (UID: \"f9d773b9-3cb1-4cba-afac-ebb1e089fac8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"
Sep 05 02:43:15.241862 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.241790    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d773b9-3cb1-4cba-afac-ebb1e089fac8-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w\" (UID: \"f9d773b9-3cb1-4cba-afac-ebb1e089fac8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"
Sep 05 02:43:15.264125 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.264095    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"]
Sep 05 02:43:15.272922 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.272891    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:15.342992 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.342960    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czqpr\" (UniqueName: \"kubernetes.io/projected/f9d773b9-3cb1-4cba-afac-ebb1e089fac8-kube-api-access-czqpr\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w\" (UID: \"f9d773b9-3cb1-4cba-afac-ebb1e089fac8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"
Sep 05 02:43:15.343176 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.343034    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d773b9-3cb1-4cba-afac-ebb1e089fac8-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w\" (UID: \"f9d773b9-3cb1-4cba-afac-ebb1e089fac8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"
Sep 05 02:43:15.346068 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.346037    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d773b9-3cb1-4cba-afac-ebb1e089fac8-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w\" (UID: \"f9d773b9-3cb1-4cba-afac-ebb1e089fac8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"
Sep 05 02:43:15.362354 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.362332    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czqpr\" (UniqueName: \"kubernetes.io/projected/f9d773b9-3cb1-4cba-afac-ebb1e089fac8-kube-api-access-czqpr\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w\" (UID: \"f9d773b9-3cb1-4cba-afac-ebb1e089fac8\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"
Sep 05 02:43:15.503976 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.503879    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"
Sep 05 02:43:15.517821 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.517796    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"]
Sep 05 02:43:15.519232 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:43:15.519193    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b55ab26_0b14_416d_8aa0_e528fca11a6a.slice/crio-65799843d2d5255f875c37f2c5d5cd5842f92f6b164b0f1472fd90bf374a5a9a WatchSource:0}: Error finding container 65799843d2d5255f875c37f2c5d5cd5842f92f6b164b0f1472fd90bf374a5a9a: Status 404 returned error can't find the container with id 65799843d2d5255f875c37f2c5d5cd5842f92f6b164b0f1472fd90bf374a5a9a
Sep 05 02:43:15.789540 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.789414    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47" event={"ID":"9b55ab26-0b14-416d-8aa0-e528fca11a6a","Type":"ContainerStarted","Data":"3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec"}
Sep 05 02:43:15.789540 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.789512    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47" event={"ID":"9b55ab26-0b14-416d-8aa0-e528fca11a6a","Type":"ContainerStarted","Data":"65799843d2d5255f875c37f2c5d5cd5842f92f6b164b0f1472fd90bf374a5a9a"}
Sep 05 02:43:15.792521 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.792498    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"]
Sep 05 02:43:15.794828 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:43:15.794802    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d773b9_3cb1_4cba_afac_ebb1e089fac8.slice/crio-5934c18b16e3e08864b70e966352e19d8ca17d1926de2b1a1e7ca4a39d4f5b3a WatchSource:0}: Error finding container 5934c18b16e3e08864b70e966352e19d8ca17d1926de2b1a1e7ca4a39d4f5b3a: Status 404 returned error can't find the container with id 5934c18b16e3e08864b70e966352e19d8ca17d1926de2b1a1e7ca4a39d4f5b3a
Sep 05 02:43:15.818946 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:15.818901    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47" podStartSLOduration=1.81888666 podStartE2EDuration="1.81888666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:43:15.816415677 +0000 UTC m=+1499.371572000" watchObservedRunningTime="2026-09-05 02:43:15.81888666 +0000 UTC m=+1499.374042968"
Sep 05 02:43:16.235496 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:16.235454    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:16.796338 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:16.796302    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w" event={"ID":"f9d773b9-3cb1-4cba-afac-ebb1e089fac8","Type":"ContainerStarted","Data":"615b231273bb8d4d7ece0b21f4b83140bb6a4c5daed74190fda7611cd349f57e"}
Sep 05 02:43:16.796537 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:16.796359    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w" event={"ID":"f9d773b9-3cb1-4cba-afac-ebb1e089fac8","Type":"ContainerStarted","Data":"5934c18b16e3e08864b70e966352e19d8ca17d1926de2b1a1e7ca4a39d4f5b3a"}
Sep 05 02:43:16.796620 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:16.796598    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"
Sep 05 02:43:16.814875 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:16.814831    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w" podStartSLOduration=1.814814974 podStartE2EDuration="1.814814974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:43:16.813504659 +0000 UTC m=+1500.368660969" watchObservedRunningTime="2026-09-05 02:43:16.814814974 +0000 UTC m=+1500.369971282"
Sep 05 02:43:16.992193 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:16.992165    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:43:16.996649 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:16.996622    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:43:21.077619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:21.077578    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 05 02:43:21.774772 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:21.774723    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:25.235432 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:25.235396    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:25.248031 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:25.248006    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:25.868733 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:25.868704    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:31.077753 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:31.077702    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 05 02:43:41.077489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:41.077369    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 05 02:43:47.804353 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:47.804317    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"
Sep 05 02:43:49.763760 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:49.763714    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"]
Sep 05 02:43:49.773918 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:49.773815    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:49.774475 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:49.774424    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"]
Sep 05 02:43:49.783773 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:43:49.783731    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:43:49.783773 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:43:49.783773 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:43:49.783773 ip-10-0-143-128 kubenswrapper[2557]:  > containerID="3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec" cmd=["/bin/sleep","15"]
Sep 05 02:43:49.784101 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:49.783787    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47" message="" err=<
Sep 05 02:43:49.784101 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:43:49.784101 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:43:49.784101 ip-10-0-143-128 kubenswrapper[2557]:  >
Sep 05 02:43:49.784101 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:43:49.783799    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:43:49.784101 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:43:49.784101 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:43:49.784101 ip-10-0-143-128 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47" podUID="9b55ab26-0b14-416d-8aa0-e528fca11a6a" containerName="main" containerID="cri-o://3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec"
Sep 05 02:43:49.784101 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:49.783847    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47" podUID="9b55ab26-0b14-416d-8aa0-e528fca11a6a" containerName="main" containerID="cri-o://3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec" gracePeriod=60
Sep 05 02:43:49.855945 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:49.855911    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:43:49.858906 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:49.858877    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47" podUID="9b55ab26-0b14-416d-8aa0-e528fca11a6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 05 02:43:50.146805 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.146779    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:50.269373 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.269341    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-dshm\") pod \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") "
Sep 05 02:43:50.269547 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.269431    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-home\") pod \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") "
Sep 05 02:43:50.269547 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.269498    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-model-cache\") pod \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") "
Sep 05 02:43:50.269667 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.269563    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2rv6\" (UniqueName: \"kubernetes.io/projected/9b55ab26-0b14-416d-8aa0-e528fca11a6a-kube-api-access-q2rv6\") pod \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") "
Sep 05 02:43:50.269667 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.269629    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b55ab26-0b14-416d-8aa0-e528fca11a6a-tls-certs\") pod \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") "
Sep 05 02:43:50.269667 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.269658    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-tmp-dir\") pod \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\" (UID: \"9b55ab26-0b14-416d-8aa0-e528fca11a6a\") "
Sep 05 02:43:50.269829 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.269766    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-home" (OuterVolumeSpecName: "home") pod "9b55ab26-0b14-416d-8aa0-e528fca11a6a" (UID: "9b55ab26-0b14-416d-8aa0-e528fca11a6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:43:50.269829 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.269804    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b55ab26-0b14-416d-8aa0-e528fca11a6a" (UID: "9b55ab26-0b14-416d-8aa0-e528fca11a6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:43:50.269996 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.269978    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:43:50.270056 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.269988    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b55ab26-0b14-416d-8aa0-e528fca11a6a" (UID: "9b55ab26-0b14-416d-8aa0-e528fca11a6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:43:50.270056 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.270001    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:43:50.271940 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.271905    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b55ab26-0b14-416d-8aa0-e528fca11a6a-kube-api-access-q2rv6" (OuterVolumeSpecName: "kube-api-access-q2rv6") pod "9b55ab26-0b14-416d-8aa0-e528fca11a6a" (UID: "9b55ab26-0b14-416d-8aa0-e528fca11a6a"). InnerVolumeSpecName "kube-api-access-q2rv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:43:50.272071 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.271954    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-dshm" (OuterVolumeSpecName: "dshm") pod "9b55ab26-0b14-416d-8aa0-e528fca11a6a" (UID: "9b55ab26-0b14-416d-8aa0-e528fca11a6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:43:50.272071 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.271968    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b55ab26-0b14-416d-8aa0-e528fca11a6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b55ab26-0b14-416d-8aa0-e528fca11a6a" (UID: "9b55ab26-0b14-416d-8aa0-e528fca11a6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:43:50.370527 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.370494    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2rv6\" (UniqueName: \"kubernetes.io/projected/9b55ab26-0b14-416d-8aa0-e528fca11a6a-kube-api-access-q2rv6\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:43:50.370527 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.370522    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b55ab26-0b14-416d-8aa0-e528fca11a6a-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:43:50.370527 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.370531    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:43:50.370527 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:50.370539    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b55ab26-0b14-416d-8aa0-e528fca11a6a-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:43:51.001759 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:51.001716    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b55ab26-0b14-416d-8aa0-e528fca11a6a" containerID="3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec" exitCode=0
Sep 05 02:43:51.002076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:51.001763    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"
Sep 05 02:43:51.002076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:51.001812    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47" event={"ID":"9b55ab26-0b14-416d-8aa0-e528fca11a6a","Type":"ContainerDied","Data":"3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec"}
Sep 05 02:43:51.002076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:51.001851    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47" event={"ID":"9b55ab26-0b14-416d-8aa0-e528fca11a6a","Type":"ContainerDied","Data":"65799843d2d5255f875c37f2c5d5cd5842f92f6b164b0f1472fd90bf374a5a9a"}
Sep 05 02:43:51.002076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:51.001872    2557 scope.go:117] "RemoveContainer" containerID="3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec"
Sep 05 02:43:51.011404 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:51.011384    2557 scope.go:117] "RemoveContainer" containerID="3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec"
Sep 05 02:43:51.011705 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:43:51.011682    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec\": container with ID starting with 3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec not found: ID does not exist" containerID="3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec"
Sep 05 02:43:51.011770 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:51.011718    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec"} err="failed to get container status \"3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec\": rpc error: code = NotFound desc = could not find container \"3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec\": container with ID starting with 3c40a99416b307b718376c352f9a682d7dede8a2271b539c649b42ad2a8992ec not found: ID does not exist"
Sep 05 02:43:51.027496 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:51.027473    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"]
Sep 05 02:43:51.035477 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:51.035452    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-bjd47"]
Sep 05 02:43:51.078336 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:51.078294    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 05 02:43:52.990760 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:52.990725    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b55ab26-0b14-416d-8aa0-e528fca11a6a" path="/var/lib/kubelet/pods/9b55ab26-0b14-416d-8aa0-e528fca11a6a/volumes"
Sep 05 02:43:57.804561 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:43:57.804517    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w" podUID="f9d773b9-3cb1-4cba-afac-ebb1e089fac8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 05 02:44:00.109838 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:00.109801    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 05 02:44:00.110208 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:44:00.109840    2557 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w" podUID="f9d773b9-3cb1-4cba-afac-ebb1e089fac8" containerName="main" containerID="cri-o://615b231273bb8d4d7ece0b21f4b83140bb6a4c5daed74190fda7611cd349f57e"
Sep 05 02:44:00.110208 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:00.109889    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w" podUID="f9d773b9-3cb1-4cba-afac-ebb1e089fac8" containerName="main" containerID="cri-o://615b231273bb8d4d7ece0b21f4b83140bb6a4c5daed74190fda7611cd349f57e" gracePeriod=50
Sep 05 02:44:00.259005 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:00.258979    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w_f9d773b9-3cb1-4cba-afac-ebb1e089fac8/main/0.log"
Sep 05 02:44:00.261242 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:00.261221    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"
Sep 05 02:44:00.363696 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:00.363609    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d773b9-3cb1-4cba-afac-ebb1e089fac8-tls-certs\") pod \"f9d773b9-3cb1-4cba-afac-ebb1e089fac8\" (UID: \"f9d773b9-3cb1-4cba-afac-ebb1e089fac8\") "
Sep 05 02:44:00.363696 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:00.363651    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czqpr\" (UniqueName: \"kubernetes.io/projected/f9d773b9-3cb1-4cba-afac-ebb1e089fac8-kube-api-access-czqpr\") pod \"f9d773b9-3cb1-4cba-afac-ebb1e089fac8\" (UID: \"f9d773b9-3cb1-4cba-afac-ebb1e089fac8\") "
Sep 05 02:44:00.365724 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:00.365697    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d773b9-3cb1-4cba-afac-ebb1e089fac8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9d773b9-3cb1-4cba-afac-ebb1e089fac8" (UID: "f9d773b9-3cb1-4cba-afac-ebb1e089fac8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:44:00.365863 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:00.365761    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d773b9-3cb1-4cba-afac-ebb1e089fac8-kube-api-access-czqpr" (OuterVolumeSpecName: "kube-api-access-czqpr") pod "f9d773b9-3cb1-4cba-afac-ebb1e089fac8" (UID: "f9d773b9-3cb1-4cba-afac-ebb1e089fac8"). InnerVolumeSpecName "kube-api-access-czqpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:44:00.464477 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:00.464424    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d773b9-3cb1-4cba-afac-ebb1e089fac8-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:44:00.464477 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:00.464474    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czqpr\" (UniqueName: \"kubernetes.io/projected/f9d773b9-3cb1-4cba-afac-ebb1e089fac8-kube-api-access-czqpr\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:44:01.055715 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:01.055687    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w_f9d773b9-3cb1-4cba-afac-ebb1e089fac8/main/0.log"
Sep 05 02:44:01.055897 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:01.055731    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9d773b9-3cb1-4cba-afac-ebb1e089fac8" containerID="615b231273bb8d4d7ece0b21f4b83140bb6a4c5daed74190fda7611cd349f57e" exitCode=1
Sep 05 02:44:01.055897 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:01.055764    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w" event={"ID":"f9d773b9-3cb1-4cba-afac-ebb1e089fac8","Type":"ContainerDied","Data":"615b231273bb8d4d7ece0b21f4b83140bb6a4c5daed74190fda7611cd349f57e"}
Sep 05 02:44:01.055897 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:01.055819    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"
Sep 05 02:44:01.055897 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:01.055862    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w" event={"ID":"f9d773b9-3cb1-4cba-afac-ebb1e089fac8","Type":"ContainerDied","Data":"5934c18b16e3e08864b70e966352e19d8ca17d1926de2b1a1e7ca4a39d4f5b3a"}
Sep 05 02:44:01.055897 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:01.055891    2557 scope.go:117] "RemoveContainer" containerID="615b231273bb8d4d7ece0b21f4b83140bb6a4c5daed74190fda7611cd349f57e"
Sep 05 02:44:01.064693 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:01.064670    2557 scope.go:117] "RemoveContainer" containerID="615b231273bb8d4d7ece0b21f4b83140bb6a4c5daed74190fda7611cd349f57e"
Sep 05 02:44:01.064974 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:44:01.064956    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615b231273bb8d4d7ece0b21f4b83140bb6a4c5daed74190fda7611cd349f57e\": container with ID starting with 615b231273bb8d4d7ece0b21f4b83140bb6a4c5daed74190fda7611cd349f57e not found: ID does not exist" containerID="615b231273bb8d4d7ece0b21f4b83140bb6a4c5daed74190fda7611cd349f57e"
Sep 05 02:44:01.065046 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:01.064981    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615b231273bb8d4d7ece0b21f4b83140bb6a4c5daed74190fda7611cd349f57e"} err="failed to get container status \"615b231273bb8d4d7ece0b21f4b83140bb6a4c5daed74190fda7611cd349f57e\": rpc error: code = NotFound desc = could not find container \"615b231273bb8d4d7ece0b21f4b83140bb6a4c5daed74190fda7611cd349f57e\": container with ID starting with 615b231273bb8d4d7ece0b21f4b83140bb6a4c5daed74190fda7611cd349f57e not found: ID does not exist"
Sep 05 02:44:01.078083 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:01.078052    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 05 02:44:01.081684 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:01.081659    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"]
Sep 05 02:44:01.087508 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:01.087487    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5x568w"]
Sep 05 02:44:02.990820 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:02.990786    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9d773b9-3cb1-4cba-afac-ebb1e089fac8" path="/var/lib/kubelet/pods/f9d773b9-3cb1-4cba-afac-ebb1e089fac8/volumes"
Sep 05 02:44:11.077637 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:11.077589    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 05 02:44:21.078335 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:21.078280    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 05 02:44:31.078186 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:31.078137    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 05 02:44:41.077622 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:41.077583    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 05 02:44:41.591974 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:41.591937    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:44:41.691023 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:41.690988    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:44:45.836478 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:45.836423    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:44:48.300291 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.300257    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"]
Sep 05 02:44:48.300933 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.300666    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b55ab26-0b14-416d-8aa0-e528fca11a6a" containerName="main"
Sep 05 02:44:48.300933 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.300680    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b55ab26-0b14-416d-8aa0-e528fca11a6a" containerName="main"
Sep 05 02:44:48.300933 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.300705    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9d773b9-3cb1-4cba-afac-ebb1e089fac8" containerName="main"
Sep 05 02:44:48.300933 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.300710    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9d773b9-3cb1-4cba-afac-ebb1e089fac8" containerName="main"
Sep 05 02:44:48.300933 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.300766    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b55ab26-0b14-416d-8aa0-e528fca11a6a" containerName="main"
Sep 05 02:44:48.300933 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.300776    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9d773b9-3cb1-4cba-afac-ebb1e089fac8" containerName="main"
Sep 05 02:44:48.304137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.304117    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.306774 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.306748    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 05 02:44:48.307498 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.307464    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 05 02:44:48.307610 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.307499    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 05 02:44:48.307610 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.307537    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-xgznt\""
Sep 05 02:44:48.316162 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.316137    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"]
Sep 05 02:44:48.367164 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.367125    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:44:48.407908 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.407870    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.408077 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.407926    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.408077 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.408028    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.408155 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.408084    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.408155 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.408116    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.408222 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.408173    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdn76\" (UniqueName: \"kubernetes.io/projected/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-kube-api-access-kdn76\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.509149 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.509113    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.509149 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.509157    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.509423 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.509182    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.509423 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.509210    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.509423 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.509262    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdn76\" (UniqueName: \"kubernetes.io/projected/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-kube-api-access-kdn76\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.509423 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.509325    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.509700 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.509619    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.509700 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.509679    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.509784 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.509724    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.511663 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.511639    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.512013 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.511989    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.518180 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.518154    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdn76\" (UniqueName: \"kubernetes.io/projected/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-kube-api-access-kdn76\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-2zp44\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.577987 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.577953    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:44:48.616007 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.615972    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:48.901345 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:48.901317    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"]
Sep 05 02:44:48.903888 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:44:48.903863    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb1cd27_b7a2_4627_a7cf_e884f4798bd9.slice/crio-21c5d19ad4c83a221d0ef8a1da66302998a53494c160e2b3e92d3b7a1b2acb91 WatchSource:0}: Error finding container 21c5d19ad4c83a221d0ef8a1da66302998a53494c160e2b3e92d3b7a1b2acb91: Status 404 returned error can't find the container with id 21c5d19ad4c83a221d0ef8a1da66302998a53494c160e2b3e92d3b7a1b2acb91
Sep 05 02:44:49.335054 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:49.335013    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44" event={"ID":"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9","Type":"ContainerStarted","Data":"4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55"}
Sep 05 02:44:49.335575 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:49.335078    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44" event={"ID":"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9","Type":"ContainerStarted","Data":"21c5d19ad4c83a221d0ef8a1da66302998a53494c160e2b3e92d3b7a1b2acb91"}
Sep 05 02:44:49.355393 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:49.355317    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44" podStartSLOduration=1.355294609 podStartE2EDuration="1.355294609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:44:49.35246186 +0000 UTC m=+1592.907618166" watchObservedRunningTime="2026-09-05 02:44:49.355294609 +0000 UTC m=+1592.910450918"
Sep 05 02:44:49.616205 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:49.616103    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:51.077638 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:51.077593    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 05 02:44:51.596056 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:51.596005    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:44:51.698848 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:51.698812    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:44:58.616164 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:58.616132    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:58.628642 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:58.628618    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:44:59.404175 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:44:59.404139    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:45:01.087475 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:01.087433    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:45:01.095202 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:01.095173    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:45:04.112477 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:04.112424    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:10.241952 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:10.241847    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"]
Sep 05 02:45:10.274168 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:10.274124    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:10.340737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:10.340706    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:21.527641 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.527599    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:21.530687 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.530663    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"]
Sep 05 02:45:21.540684 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:45:21.540638    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:45:21.540684 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:45:21.540684 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:45:21.540684 ip-10-0-143-128 kubenswrapper[2557]:  > containerID="4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55" cmd=["/bin/sleep","15"]
Sep 05 02:45:21.540952 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.540697    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44" message="" err=<
Sep 05 02:45:21.540952 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:45:21.540952 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:45:21.540952 ip-10-0-143-128 kubenswrapper[2557]:  >
Sep 05 02:45:21.540952 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:45:21.540715    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:45:21.540952 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:45:21.540952 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:45:21.540952 ip-10-0-143-128 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44" podUID="4cb1cd27-b7a2-4627-a7cf-e884f4798bd9" containerName="main" containerID="cri-o://4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55"
Sep 05 02:45:21.540952 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.540753    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44" podUID="4cb1cd27-b7a2-4627-a7cf-e884f4798bd9" containerName="main" containerID="cri-o://4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55" gracePeriod=60
Sep 05 02:45:21.891261 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.891235    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:45:21.923223 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.923199    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-home\") pod \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") "
Sep 05 02:45:21.923397 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.923244    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-tls-certs\") pod \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") "
Sep 05 02:45:21.923397 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.923288    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-dshm\") pod \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") "
Sep 05 02:45:21.923397 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.923318    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-model-cache\") pod \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") "
Sep 05 02:45:21.923397 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.923386    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-tmp-dir\") pod \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") "
Sep 05 02:45:21.923674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.923486    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdn76\" (UniqueName: \"kubernetes.io/projected/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-kube-api-access-kdn76\") pod \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\" (UID: \"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9\") "
Sep 05 02:45:21.923859 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.923799    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-home" (OuterVolumeSpecName: "home") pod "4cb1cd27-b7a2-4627-a7cf-e884f4798bd9" (UID: "4cb1cd27-b7a2-4627-a7cf-e884f4798bd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:45:21.923954 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.923874    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4cb1cd27-b7a2-4627-a7cf-e884f4798bd9" (UID: "4cb1cd27-b7a2-4627-a7cf-e884f4798bd9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:45:21.923954 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.923835    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-model-cache" (OuterVolumeSpecName: "model-cache") pod "4cb1cd27-b7a2-4627-a7cf-e884f4798bd9" (UID: "4cb1cd27-b7a2-4627-a7cf-e884f4798bd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:45:21.925822 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.925784    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-kube-api-access-kdn76" (OuterVolumeSpecName: "kube-api-access-kdn76") pod "4cb1cd27-b7a2-4627-a7cf-e884f4798bd9" (UID: "4cb1cd27-b7a2-4627-a7cf-e884f4798bd9"). InnerVolumeSpecName "kube-api-access-kdn76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:45:21.925822 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.925791    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4cb1cd27-b7a2-4627-a7cf-e884f4798bd9" (UID: "4cb1cd27-b7a2-4627-a7cf-e884f4798bd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:45:21.925950 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:21.925794    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-dshm" (OuterVolumeSpecName: "dshm") pod "4cb1cd27-b7a2-4627-a7cf-e884f4798bd9" (UID: "4cb1cd27-b7a2-4627-a7cf-e884f4798bd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:45:22.024426 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.024387    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:45:22.024426 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.024419    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:45:22.024426 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.024428    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:45:22.024697 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.024463    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdn76\" (UniqueName: \"kubernetes.io/projected/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-kube-api-access-kdn76\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:45:22.024697 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.024474    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:45:22.024697 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.024483    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:45:22.522197 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.522165    2557 generic.go:356] "Generic (PLEG): container finished" podID="4cb1cd27-b7a2-4627-a7cf-e884f4798bd9" containerID="4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55" exitCode=0
Sep 05 02:45:22.522372 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.522213    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"
Sep 05 02:45:22.522372 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.522223    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44" event={"ID":"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9","Type":"ContainerDied","Data":"4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55"}
Sep 05 02:45:22.522372 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.522271    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44" event={"ID":"4cb1cd27-b7a2-4627-a7cf-e884f4798bd9","Type":"ContainerDied","Data":"21c5d19ad4c83a221d0ef8a1da66302998a53494c160e2b3e92d3b7a1b2acb91"}
Sep 05 02:45:22.522372 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.522294    2557 scope.go:117] "RemoveContainer" containerID="4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55"
Sep 05 02:45:22.532495 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.532477    2557 scope.go:117] "RemoveContainer" containerID="4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55"
Sep 05 02:45:22.532810 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:45:22.532753    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55\": container with ID starting with 4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55 not found: ID does not exist" containerID="4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55"
Sep 05 02:45:22.532810 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.532778    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55"} err="failed to get container status \"4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55\": rpc error: code = NotFound desc = could not find container \"4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55\": container with ID starting with 4b9d87658f65a8a50e7a3b0e5435d6c3e2ae9844f29c97ee109179995f983e55 not found: ID does not exist"
Sep 05 02:45:22.544128 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.544106    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"]
Sep 05 02:45:22.547727 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.547704    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-2zp44"]
Sep 05 02:45:22.990559 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:22.990528    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cb1cd27-b7a2-4627-a7cf-e884f4798bd9" path="/var/lib/kubelet/pods/4cb1cd27-b7a2-4627-a7cf-e884f4798bd9/volumes"
Sep 05 02:45:24.321978 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:24.321941    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:25.271586 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:25.271539    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main" containerID="cri-o://3e9dfd8944af1ca22ef91d1ed3c9f718d5be668e46e47fe985b0f27f6b0b8276" gracePeriod=45
Sep 05 02:45:26.030096 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.030063    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:45:26.052062 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.051984    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-tls-certs\") pod \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") "
Sep 05 02:45:26.052062 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.052054    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-kserve-provision-location\") pod \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") "
Sep 05 02:45:26.052254 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.052185    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-cabundle-cert\") pod \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") "
Sep 05 02:45:26.052254 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.052241    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv5dj\" (UniqueName: \"kubernetes.io/projected/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-kube-api-access-sv5dj\") pod \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") "
Sep 05 02:45:26.052363 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.052311    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-tmp-dir\") pod \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") "
Sep 05 02:45:26.052363 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.052348    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-home\") pod \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") "
Sep 05 02:45:26.052485 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.052406    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-model-cache\") pod \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") "
Sep 05 02:45:26.052485 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.052469    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-dshm\") pod \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\" (UID: \"6c4649cd-5bda-470e-b2f0-8a29739dc8f5\") "
Sep 05 02:45:26.053272 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.053240    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-home" (OuterVolumeSpecName: "home") pod "6c4649cd-5bda-470e-b2f0-8a29739dc8f5" (UID: "6c4649cd-5bda-470e-b2f0-8a29739dc8f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:45:26.053734 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.053705    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c4649cd-5bda-470e-b2f0-8a29739dc8f5" (UID: "6c4649cd-5bda-470e-b2f0-8a29739dc8f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:45:26.054002 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.053976    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6c4649cd-5bda-470e-b2f0-8a29739dc8f5" (UID: "6c4649cd-5bda-470e-b2f0-8a29739dc8f5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:45:26.055034 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.055006    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c4649cd-5bda-470e-b2f0-8a29739dc8f5" (UID: "6c4649cd-5bda-470e-b2f0-8a29739dc8f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:45:26.055189 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.055163    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-kube-api-access-sv5dj" (OuterVolumeSpecName: "kube-api-access-sv5dj") pod "6c4649cd-5bda-470e-b2f0-8a29739dc8f5" (UID: "6c4649cd-5bda-470e-b2f0-8a29739dc8f5"). InnerVolumeSpecName "kube-api-access-sv5dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:45:26.057209 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.056937    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-dshm" (OuterVolumeSpecName: "dshm") pod "6c4649cd-5bda-470e-b2f0-8a29739dc8f5" (UID: "6c4649cd-5bda-470e-b2f0-8a29739dc8f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:45:26.074928 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.074893    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c4649cd-5bda-470e-b2f0-8a29739dc8f5" (UID: "6c4649cd-5bda-470e-b2f0-8a29739dc8f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:45:26.089855 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.089811    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c4649cd-5bda-470e-b2f0-8a29739dc8f5" (UID: "6c4649cd-5bda-470e-b2f0-8a29739dc8f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:45:26.153888 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.153844    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:45:26.153888 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.153877    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:45:26.153888 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.153890    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:45:26.154139 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.153901    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:45:26.154139 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.153912    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:45:26.154139 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.153924    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:45:26.154139 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.153935    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:45:26.154139 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.153949    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv5dj\" (UniqueName: \"kubernetes.io/projected/6c4649cd-5bda-470e-b2f0-8a29739dc8f5-kube-api-access-sv5dj\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:45:26.547401 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.547363    2557 generic.go:356] "Generic (PLEG): container finished" podID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerID="3e9dfd8944af1ca22ef91d1ed3c9f718d5be668e46e47fe985b0f27f6b0b8276" exitCode=0
Sep 05 02:45:26.547579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.547411    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"
Sep 05 02:45:26.547579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.547431    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" event={"ID":"6c4649cd-5bda-470e-b2f0-8a29739dc8f5","Type":"ContainerDied","Data":"3e9dfd8944af1ca22ef91d1ed3c9f718d5be668e46e47fe985b0f27f6b0b8276"}
Sep 05 02:45:26.547579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.547520    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm" event={"ID":"6c4649cd-5bda-470e-b2f0-8a29739dc8f5","Type":"ContainerDied","Data":"18c16af2095cae5278bcac2181b8d9bd5137666a565b637f865a629f5fbe3b61"}
Sep 05 02:45:26.547579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.547542    2557 scope.go:117] "RemoveContainer" containerID="3e9dfd8944af1ca22ef91d1ed3c9f718d5be668e46e47fe985b0f27f6b0b8276"
Sep 05 02:45:26.556903 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.556887    2557 scope.go:117] "RemoveContainer" containerID="fcc35ada4d7a6786873d1959ff230d69194040447dcaa16edff821ee81937a3b"
Sep 05 02:45:26.564906 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.564888    2557 scope.go:117] "RemoveContainer" containerID="3e9dfd8944af1ca22ef91d1ed3c9f718d5be668e46e47fe985b0f27f6b0b8276"
Sep 05 02:45:26.565157 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:45:26.565135    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9dfd8944af1ca22ef91d1ed3c9f718d5be668e46e47fe985b0f27f6b0b8276\": container with ID starting with 3e9dfd8944af1ca22ef91d1ed3c9f718d5be668e46e47fe985b0f27f6b0b8276 not found: ID does not exist" containerID="3e9dfd8944af1ca22ef91d1ed3c9f718d5be668e46e47fe985b0f27f6b0b8276"
Sep 05 02:45:26.565216 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.565163    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9dfd8944af1ca22ef91d1ed3c9f718d5be668e46e47fe985b0f27f6b0b8276"} err="failed to get container status \"3e9dfd8944af1ca22ef91d1ed3c9f718d5be668e46e47fe985b0f27f6b0b8276\": rpc error: code = NotFound desc = could not find container \"3e9dfd8944af1ca22ef91d1ed3c9f718d5be668e46e47fe985b0f27f6b0b8276\": container with ID starting with 3e9dfd8944af1ca22ef91d1ed3c9f718d5be668e46e47fe985b0f27f6b0b8276 not found: ID does not exist"
Sep 05 02:45:26.565216 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.565179    2557 scope.go:117] "RemoveContainer" containerID="fcc35ada4d7a6786873d1959ff230d69194040447dcaa16edff821ee81937a3b"
Sep 05 02:45:26.565449 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:45:26.565417    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc35ada4d7a6786873d1959ff230d69194040447dcaa16edff821ee81937a3b\": container with ID starting with fcc35ada4d7a6786873d1959ff230d69194040447dcaa16edff821ee81937a3b not found: ID does not exist" containerID="fcc35ada4d7a6786873d1959ff230d69194040447dcaa16edff821ee81937a3b"
Sep 05 02:45:26.565511 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.565460    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc35ada4d7a6786873d1959ff230d69194040447dcaa16edff821ee81937a3b"} err="failed to get container status \"fcc35ada4d7a6786873d1959ff230d69194040447dcaa16edff821ee81937a3b\": rpc error: code = NotFound desc = could not find container \"fcc35ada4d7a6786873d1959ff230d69194040447dcaa16edff821ee81937a3b\": container with ID starting with fcc35ada4d7a6786873d1959ff230d69194040447dcaa16edff821ee81937a3b not found: ID does not exist"
Sep 05 02:45:26.572721 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.572656    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"]
Sep 05 02:45:26.577375 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.577352    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86cf5c7489-27xnm"]
Sep 05 02:45:26.990838 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:26.990803    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" path="/var/lib/kubelet/pods/6c4649cd-5bda-470e-b2f0-8a29739dc8f5/volumes"
Sep 05 02:45:47.190886 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:47.190837    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:45:47.292192 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:45:47.292157    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:00.367212 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:00.367174    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:00.467474 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:00.467415    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:00.570808 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:00.570774    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:04.557496 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:04.557457    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:06.815141 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:06.815104    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:10.909671 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.909631    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"]
Sep 05 02:46:10.910179 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.910155    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main"
Sep 05 02:46:10.910230 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.910183    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main"
Sep 05 02:46:10.910230 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.910213    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="storage-initializer"
Sep 05 02:46:10.910230 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.910223    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="storage-initializer"
Sep 05 02:46:10.910345 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.910238    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb1cd27-b7a2-4627-a7cf-e884f4798bd9" containerName="main"
Sep 05 02:46:10.910345 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.910247    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb1cd27-b7a2-4627-a7cf-e884f4798bd9" containerName="main"
Sep 05 02:46:10.910345 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.910333    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c4649cd-5bda-470e-b2f0-8a29739dc8f5" containerName="main"
Sep 05 02:46:10.910461 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.910351    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb1cd27-b7a2-4627-a7cf-e884f4798bd9" containerName="main"
Sep 05 02:46:10.913629 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.913604    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:10.916737 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.916717    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 05 02:46:10.917651 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.917631    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 05 02:46:10.917874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.917854    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 05 02:46:10.917952 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.917934    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-xllc5\""
Sep 05 02:46:10.930110 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.930082    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"]
Sep 05 02:46:10.940652 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:10.940624    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:11.030892 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.030859    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.031062 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.030900    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-dshm\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.031062 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.030934    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-home\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.031062 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.031003    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fctvf\" (UniqueName: \"kubernetes.io/projected/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-kube-api-access-fctvf\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.031062 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.031026    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.031062 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.031042    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.132378 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.132345    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.132573 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.132430    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-dshm\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.132573 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.132539    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-home\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.132702 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.132578    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fctvf\" (UniqueName: \"kubernetes.io/projected/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-kube-api-access-fctvf\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.132702 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.132625    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.132702 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.132653    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.133070 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.133049    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.133165 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.133100    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.133165 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.133133    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-home\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.134945 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.134922    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-dshm\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.135065 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.135042    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.141846 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.141815    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fctvf\" (UniqueName: \"kubernetes.io/projected/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-kube-api-access-fctvf\") pod \"canary-v2-kserve-7f65dbbccc-tbftd\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.223863 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.223796    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:11.528627 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:46:11.528582    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b86667b_243c_49ec_adc7_0bb1d5bc6b08.slice/crio-7dcfc6278128f7a98f57435b92ea7d6319c8332623ad0da8494e5f6de14712f6 WatchSource:0}: Error finding container 7dcfc6278128f7a98f57435b92ea7d6319c8332623ad0da8494e5f6de14712f6: Status 404 returned error can't find the container with id 7dcfc6278128f7a98f57435b92ea7d6319c8332623ad0da8494e5f6de14712f6
Sep 05 02:46:11.528765 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.528733    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"]
Sep 05 02:46:11.531464 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.531419    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:46:11.592372 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.592337    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:11.738579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.738540    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:11.788107 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:11.788018    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd" event={"ID":"6b86667b-243c-49ec-adc7-0bb1d5bc6b08","Type":"ContainerStarted","Data":"7dcfc6278128f7a98f57435b92ea7d6319c8332623ad0da8494e5f6de14712f6"}
Sep 05 02:46:12.800726 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:12.800636    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd" event={"ID":"6b86667b-243c-49ec-adc7-0bb1d5bc6b08","Type":"ContainerStarted","Data":"b8fc2fc993ccc3ff1632c4abf42c59c997d4f36efb0c56cc8462c46974bed47d"}
Sep 05 02:46:12.823405 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:12.823357    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd" podStartSLOduration=1.8607959630000002 podStartE2EDuration="2.823341628s" totalImagesPullingTime="962.545665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:46:12.822589252 +0000 UTC m=+1676.377745563" watchObservedRunningTime="2026-09-05 02:46:12.823341628 +0000 UTC m=+1676.378497937"
Sep 05 02:46:13.224682 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:13.224638    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:13.515918 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:13.515832    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:13.615977 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:13.615941    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:17.816357 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:17.816301    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:21.224599 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:21.224564    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:21.236922 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:21.236898    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:21.859562 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:21.859534    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:46:23.056414 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:23.056373    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:23.381722 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:23.381683    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:23.447988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:23.447954    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:35.626803 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.626763    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"]
Sep 05 02:46:35.630524 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.630498    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.633623 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.633593    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 05 02:46:35.633734 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.633673    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 05 02:46:35.636642 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.636415    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-tws9t\""
Sep 05 02:46:35.636865 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.636806    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-zxkvm\""
Sep 05 02:46:35.637055 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.637035    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 05 02:46:35.641844 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.641823    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"]
Sep 05 02:46:35.697282 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.697251    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:35.744973 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.744946    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.745088 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.744977    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.745088 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.744994    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.745088 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.745017    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/461d2f4d-16f9-41a1-835c-d0f726574833-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.745193 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.745079    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjj8\" (UniqueName: \"kubernetes.io/projected/461d2f4d-16f9-41a1-835c-d0f726574833-kube-api-access-sfjj8\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.745193 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.745146    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.845970 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.845933    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.845970 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.845975    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.846217 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.845998    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/461d2f4d-16f9-41a1-835c-d0f726574833-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.846217 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.846025    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjj8\" (UniqueName: \"kubernetes.io/projected/461d2f4d-16f9-41a1-835c-d0f726574833-kube-api-access-sfjj8\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.846217 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.846083    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.846217 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.846156    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.846461 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.846419    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.846588 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.846570    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.846639 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.846618    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.848471 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.848432    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.848572 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.848528    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/461d2f4d-16f9-41a1-835c-d0f726574833-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.855050 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.855025    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjj8\" (UniqueName: \"kubernetes.io/projected/461d2f4d-16f9-41a1-835c-d0f726574833-kube-api-access-sfjj8\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:35.906847 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.906760    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:46:35.941271 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:35.941240    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:36.219646 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:36.219551    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"]
Sep 05 02:46:36.221528 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:46:36.221499    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461d2f4d_16f9_41a1_835c_d0f726574833.slice/crio-9ac45f5be2b54f4c88289801a4c37c7baa7f0b6472627e33b88903e01d8c9960 WatchSource:0}: Error finding container 9ac45f5be2b54f4c88289801a4c37c7baa7f0b6472627e33b88903e01d8c9960: Status 404 returned error can't find the container with id 9ac45f5be2b54f4c88289801a4c37c7baa7f0b6472627e33b88903e01d8c9960
Sep 05 02:46:36.931918 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:36.931885    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk" event={"ID":"461d2f4d-16f9-41a1-835c-d0f726574833","Type":"ContainerStarted","Data":"9ac45f5be2b54f4c88289801a4c37c7baa7f0b6472627e33b88903e01d8c9960"}
Sep 05 02:46:38.948803 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:38.948695    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk" event={"ID":"461d2f4d-16f9-41a1-835c-d0f726574833","Type":"ContainerStarted","Data":"ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a"}
Sep 05 02:46:38.949246 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:38.948932    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:39.956042 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:39.956006    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk" event={"ID":"461d2f4d-16f9-41a1-835c-d0f726574833","Type":"ContainerStarted","Data":"196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9"}
Sep 05 02:46:39.975416 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:39.975368    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk" podStartSLOduration=2.629054665 podStartE2EDuration="4.975352228s" totalImagesPullingTime="2.346297563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:46:39.974178321 +0000 UTC m=+1703.529334629" watchObservedRunningTime="2026-09-05 02:46:39.975352228 +0000 UTC m=+1703.530508563"
Sep 05 02:46:40.941404 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:40.941365    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:45.941956 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:45.941919    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:45.954785 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:45.954758    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:46.001416 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:46.001381    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:46:57.008290 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:46:57.008256    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:47:02.817006 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:47:02.816963    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:47:57.521377 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:47:57.521326    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:47:57.599227 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:47:57.599187    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"]
Sep 05 02:47:57.622361 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:47:57.622326    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:47:57.721821 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:47:57.721783    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:47:58.218247 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:47:58.218212    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:02.595773 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:02.595736    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:06.606172 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.606135    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"]
Sep 05 02:48:06.609478 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.609459    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.612514 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.612494    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 05 02:48:06.612629 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.612497    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 05 02:48:06.612629 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.612530    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 05 02:48:06.612629 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.612590    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-cxvgs\""
Sep 05 02:48:06.622359 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.622335    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"]
Sep 05 02:48:06.633274 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.633244    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-dshm\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.633402 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.633279    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-model-cache\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.633402 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.633320    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-home\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.633402 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.633354    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.633584 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.633419    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80ffc3f4-a7b9-452e-9381-937c8babac28-tls-certs\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.633584 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.633462    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dgf\" (UniqueName: \"kubernetes.io/projected/80ffc3f4-a7b9-452e-9381-937c8babac28-kube-api-access-h2dgf\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.634867 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.634840    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:06.734886 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.734847    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-home\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.734886 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.734889    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.735099 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.734912    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80ffc3f4-a7b9-452e-9381-937c8babac28-tls-certs\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.735099 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.734943    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dgf\" (UniqueName: \"kubernetes.io/projected/80ffc3f4-a7b9-452e-9381-937c8babac28-kube-api-access-h2dgf\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.735099 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.735003    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-dshm\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.735099 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.735025    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-model-cache\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.735461 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.735406    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-home\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.735591 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.735418    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-model-cache\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.735591 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.735490    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.737306 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.737285    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-dshm\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.737726 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.737706    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80ffc3f4-a7b9-452e-9381-937c8babac28-tls-certs\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.743815 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.743785    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dgf\" (UniqueName: \"kubernetes.io/projected/80ffc3f4-a7b9-452e-9381-937c8babac28-kube-api-access-h2dgf\") pod \"test-div-v1-kserve-fb566d667-zfc5z\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:06.919917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:06.919830    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:07.206978 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:07.206945    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"]
Sep 05 02:48:07.209556 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:48:07.209512    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ffc3f4_a7b9_452e_9381_937c8babac28.slice/crio-4f167090148ca3e65ab94101a15151f7f657ac8aa91097a8ce072a502feb9385 WatchSource:0}: Error finding container 4f167090148ca3e65ab94101a15151f7f657ac8aa91097a8ce072a502feb9385: Status 404 returned error can't find the container with id 4f167090148ca3e65ab94101a15151f7f657ac8aa91097a8ce072a502feb9385
Sep 05 02:48:07.420348 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:07.420309    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z" event={"ID":"80ffc3f4-a7b9-452e-9381-937c8babac28","Type":"ContainerStarted","Data":"83f9057a86303ada2df4a35ca4e7f34546a5d8b1922bd67dbeceee356126fbb9"}
Sep 05 02:48:07.420566 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:07.420400    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z" event={"ID":"80ffc3f4-a7b9-452e-9381-937c8babac28","Type":"ContainerStarted","Data":"4f167090148ca3e65ab94101a15151f7f657ac8aa91097a8ce072a502feb9385"}
Sep 05 02:48:07.437243 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:07.437195    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z" podStartSLOduration=1.4371817230000001 podStartE2EDuration="1.437181723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:48:07.435964073 +0000 UTC m=+1790.991120382" watchObservedRunningTime="2026-09-05 02:48:07.437181723 +0000 UTC m=+1790.992338032"
Sep 05 02:48:07.920018 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:07.919972    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:12.396212 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.396134    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"]
Sep 05 02:48:12.402463 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.402412    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.405013 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.404987    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 05 02:48:12.411374 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.411333    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"]
Sep 05 02:48:12.426277 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.426246    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:12.487091 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.487045    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzwvj\" (UniqueName: \"kubernetes.io/projected/6edd4dde-b03e-4b9f-ab4f-b91199400864-kube-api-access-fzwvj\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.487291 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.487103    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.487291 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.487147    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-home\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.487291 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.487169    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.487291 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.487241    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edd4dde-b03e-4b9f-ab4f-b91199400864-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.487569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.487336    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.588305 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.588262    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-home\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.588305 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.588300    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.588589 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.588319    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edd4dde-b03e-4b9f-ab4f-b91199400864-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.588589 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.588373    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.588589 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.588411    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzwvj\" (UniqueName: \"kubernetes.io/projected/6edd4dde-b03e-4b9f-ab4f-b91199400864-kube-api-access-fzwvj\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.588589 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.588470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.588807 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.588777    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-home\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.588904 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.588883    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.589005 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.588989    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.590806 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.590777    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.591236 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.591212    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edd4dde-b03e-4b9f-ab4f-b91199400864-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.597141 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.597119    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzwvj\" (UniqueName: \"kubernetes.io/projected/6edd4dde-b03e-4b9f-ab4f-b91199400864-kube-api-access-fzwvj\") pod \"test-div-v2-kserve-5f6fdb6b77-868j8\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.619653 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.619623    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd" podUID="6b86667b-243c-49ec-adc7-0bb1d5bc6b08" containerName="main" containerID="cri-o://b8fc2fc993ccc3ff1632c4abf42c59c997d4f36efb0c56cc8462c46974bed47d" gracePeriod=45
Sep 05 02:48:12.715517 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.715419    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:12.889376 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.889349    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:48:12.996475 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.995473    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-home\") pod \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") "
Sep 05 02:48:12.996475 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.995540    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fctvf\" (UniqueName: \"kubernetes.io/projected/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-kube-api-access-fctvf\") pod \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") "
Sep 05 02:48:12.996475 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.995600    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-dshm\") pod \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") "
Sep 05 02:48:12.996475 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.995646    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-model-cache\") pod \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") "
Sep 05 02:48:12.996475 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.995701    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-tmp-dir\") pod \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") "
Sep 05 02:48:12.996475 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.995746    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-tls-certs\") pod \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\" (UID: \"6b86667b-243c-49ec-adc7-0bb1d5bc6b08\") "
Sep 05 02:48:12.997393 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.997208    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-home" (OuterVolumeSpecName: "home") pod "6b86667b-243c-49ec-adc7-0bb1d5bc6b08" (UID: "6b86667b-243c-49ec-adc7-0bb1d5bc6b08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:12.997828 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:12.997571    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b86667b-243c-49ec-adc7-0bb1d5bc6b08" (UID: "6b86667b-243c-49ec-adc7-0bb1d5bc6b08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:13.000727 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.000569    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b86667b-243c-49ec-adc7-0bb1d5bc6b08" (UID: "6b86667b-243c-49ec-adc7-0bb1d5bc6b08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:13.000727 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.000590    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-kube-api-access-fctvf" (OuterVolumeSpecName: "kube-api-access-fctvf") pod "6b86667b-243c-49ec-adc7-0bb1d5bc6b08" (UID: "6b86667b-243c-49ec-adc7-0bb1d5bc6b08"). InnerVolumeSpecName "kube-api-access-fctvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:48:13.001209 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.001075    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b86667b-243c-49ec-adc7-0bb1d5bc6b08" (UID: "6b86667b-243c-49ec-adc7-0bb1d5bc6b08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:48:13.002218 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.002192    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-dshm" (OuterVolumeSpecName: "dshm") pod "6b86667b-243c-49ec-adc7-0bb1d5bc6b08" (UID: "6b86667b-243c-49ec-adc7-0bb1d5bc6b08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:13.003015 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.002989    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"]
Sep 05 02:48:13.004222 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:48:13.004192    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edd4dde_b03e_4b9f_ab4f_b91199400864.slice/crio-e7873ba9504c3a476fa1823a5389ad9e5bd8dab4b07aa07fa75bce6ffbfb6b92 WatchSource:0}: Error finding container e7873ba9504c3a476fa1823a5389ad9e5bd8dab4b07aa07fa75bce6ffbfb6b92: Status 404 returned error can't find the container with id e7873ba9504c3a476fa1823a5389ad9e5bd8dab4b07aa07fa75bce6ffbfb6b92
Sep 05 02:48:13.097453 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.097410    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:13.097453 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.097454    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:13.097453 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.097465    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:13.097757 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.097473    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:13.097757 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.097482    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:13.097757 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.097490    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fctvf\" (UniqueName: \"kubernetes.io/projected/6b86667b-243c-49ec-adc7-0bb1d5bc6b08-kube-api-access-fctvf\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:13.453695 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.453661    2557 generic.go:356] "Generic (PLEG): container finished" podID="6b86667b-243c-49ec-adc7-0bb1d5bc6b08" containerID="b8fc2fc993ccc3ff1632c4abf42c59c997d4f36efb0c56cc8462c46974bed47d" exitCode=0
Sep 05 02:48:13.454098 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.453723    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"
Sep 05 02:48:13.454098 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.453742    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd" event={"ID":"6b86667b-243c-49ec-adc7-0bb1d5bc6b08","Type":"ContainerDied","Data":"b8fc2fc993ccc3ff1632c4abf42c59c997d4f36efb0c56cc8462c46974bed47d"}
Sep 05 02:48:13.454098 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.453793    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd" event={"ID":"6b86667b-243c-49ec-adc7-0bb1d5bc6b08","Type":"ContainerDied","Data":"7dcfc6278128f7a98f57435b92ea7d6319c8332623ad0da8494e5f6de14712f6"}
Sep 05 02:48:13.454098 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.453834    2557 scope.go:117] "RemoveContainer" containerID="b8fc2fc993ccc3ff1632c4abf42c59c997d4f36efb0c56cc8462c46974bed47d"
Sep 05 02:48:13.455623 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.455597    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8" event={"ID":"6edd4dde-b03e-4b9f-ab4f-b91199400864","Type":"ContainerStarted","Data":"75ab052fdd6093fca272f0b716391029b65fcad253cb35cf464ca7734d832a78"}
Sep 05 02:48:13.455717 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.455654    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8" event={"ID":"6edd4dde-b03e-4b9f-ab4f-b91199400864","Type":"ContainerStarted","Data":"e7873ba9504c3a476fa1823a5389ad9e5bd8dab4b07aa07fa75bce6ffbfb6b92"}
Sep 05 02:48:13.464409 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.464393    2557 scope.go:117] "RemoveContainer" containerID="b8fc2fc993ccc3ff1632c4abf42c59c997d4f36efb0c56cc8462c46974bed47d"
Sep 05 02:48:13.464740 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:48:13.464718    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8fc2fc993ccc3ff1632c4abf42c59c997d4f36efb0c56cc8462c46974bed47d\": container with ID starting with b8fc2fc993ccc3ff1632c4abf42c59c997d4f36efb0c56cc8462c46974bed47d not found: ID does not exist" containerID="b8fc2fc993ccc3ff1632c4abf42c59c997d4f36efb0c56cc8462c46974bed47d"
Sep 05 02:48:13.464795 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.464750    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8fc2fc993ccc3ff1632c4abf42c59c997d4f36efb0c56cc8462c46974bed47d"} err="failed to get container status \"b8fc2fc993ccc3ff1632c4abf42c59c997d4f36efb0c56cc8462c46974bed47d\": rpc error: code = NotFound desc = could not find container \"b8fc2fc993ccc3ff1632c4abf42c59c997d4f36efb0c56cc8462c46974bed47d\": container with ID starting with b8fc2fc993ccc3ff1632c4abf42c59c997d4f36efb0c56cc8462c46974bed47d not found: ID does not exist"
Sep 05 02:48:13.478225 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.478177    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8" podStartSLOduration=1.478165066 podStartE2EDuration="1.478165066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:48:13.477090356 +0000 UTC m=+1797.032246665" watchObservedRunningTime="2026-09-05 02:48:13.478165066 +0000 UTC m=+1797.033321374"
Sep 05 02:48:13.490099 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.490074    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"]
Sep 05 02:48:13.495288 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.495266    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd"]
Sep 05 02:48:13.716022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.715931    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:13.849489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:13.849411    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-tbftd" podUID="6b86667b-243c-49ec-adc7-0bb1d5bc6b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 05 02:48:14.991522 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:14.991487    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b86667b-243c-49ec-adc7-0bb1d5bc6b08" path="/var/lib/kubelet/pods/6b86667b-243c-49ec-adc7-0bb1d5bc6b08/volumes"
Sep 05 02:48:16.920966 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:16.920921    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:16.939724 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:16.939690    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:17.046460 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:17.046414    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:48:17.051221 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:17.051196    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:48:17.492086 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:17.492050    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:22.716328 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:22.716291    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:22.728790 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:22.728761    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:23.523544 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:23.523510    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:34.714880 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:34.714843    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:34.726766 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:34.726732    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"]
Sep 05 02:48:34.736929 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:48:34.736893    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:48:34.736929 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:48:34.736929 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:48:34.736929 ip-10-0-143-128 kubenswrapper[2557]:  > containerID="196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9" cmd=["/bin/sleep","15"]
Sep 05 02:48:34.737249 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:34.736943    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk" message="" err=<
Sep 05 02:48:34.737249 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:48:34.737249 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:48:34.737249 ip-10-0-143-128 kubenswrapper[2557]:  >
Sep 05 02:48:34.737249 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:48:34.736956    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:48:34.737249 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:48:34.737249 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:48:34.737249 ip-10-0-143-128 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk" podUID="461d2f4d-16f9-41a1-835c-d0f726574833" containerName="main" containerID="cri-o://196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9"
Sep 05 02:48:34.737249 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:34.737001    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk" podUID="461d2f4d-16f9-41a1-835c-d0f726574833" containerName="main" containerID="cri-o://196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9" gracePeriod=60
Sep 05 02:48:34.781336 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:34.781300    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:34.954921 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:34.954877    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk" podUID="461d2f4d-16f9-41a1-835c-d0f726574833" containerName="llm-d-routing-sidecar" containerID="cri-o://ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a" gracePeriod=60
Sep 05 02:48:34.991060 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:34.991022    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk" podUID="461d2f4d-16f9-41a1-835c-d0f726574833" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 05 02:48:35.208939 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.208911    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:48:35.308006 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.307915    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-model-cache\") pod \"461d2f4d-16f9-41a1-835c-d0f726574833\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") "
Sep 05 02:48:35.308006 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.307963    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-dshm\") pod \"461d2f4d-16f9-41a1-835c-d0f726574833\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") "
Sep 05 02:48:35.308006 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.307992    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-tmp-dir\") pod \"461d2f4d-16f9-41a1-835c-d0f726574833\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") "
Sep 05 02:48:35.308006 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.308009    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfjj8\" (UniqueName: \"kubernetes.io/projected/461d2f4d-16f9-41a1-835c-d0f726574833-kube-api-access-sfjj8\") pod \"461d2f4d-16f9-41a1-835c-d0f726574833\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") "
Sep 05 02:48:35.308345 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.308054    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/461d2f4d-16f9-41a1-835c-d0f726574833-tls-certs\") pod \"461d2f4d-16f9-41a1-835c-d0f726574833\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") "
Sep 05 02:48:35.308345 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.308085    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-home\") pod \"461d2f4d-16f9-41a1-835c-d0f726574833\" (UID: \"461d2f4d-16f9-41a1-835c-d0f726574833\") "
Sep 05 02:48:35.308345 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.308242    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "461d2f4d-16f9-41a1-835c-d0f726574833" (UID: "461d2f4d-16f9-41a1-835c-d0f726574833"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:35.308345 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.308255    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-model-cache" (OuterVolumeSpecName: "model-cache") pod "461d2f4d-16f9-41a1-835c-d0f726574833" (UID: "461d2f4d-16f9-41a1-835c-d0f726574833"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:35.308538 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.308426    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:35.308538 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.308460    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:35.308538 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.308476    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-home" (OuterVolumeSpecName: "home") pod "461d2f4d-16f9-41a1-835c-d0f726574833" (UID: "461d2f4d-16f9-41a1-835c-d0f726574833"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:35.310269 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.310241    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/461d2f4d-16f9-41a1-835c-d0f726574833-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "461d2f4d-16f9-41a1-835c-d0f726574833" (UID: "461d2f4d-16f9-41a1-835c-d0f726574833"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:48:35.310269 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.310250    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-dshm" (OuterVolumeSpecName: "dshm") pod "461d2f4d-16f9-41a1-835c-d0f726574833" (UID: "461d2f4d-16f9-41a1-835c-d0f726574833"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:35.310432 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.310272    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461d2f4d-16f9-41a1-835c-d0f726574833-kube-api-access-sfjj8" (OuterVolumeSpecName: "kube-api-access-sfjj8") pod "461d2f4d-16f9-41a1-835c-d0f726574833" (UID: "461d2f4d-16f9-41a1-835c-d0f726574833"). InnerVolumeSpecName "kube-api-access-sfjj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:48:35.409517 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.409480    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:35.409517 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.409513    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfjj8\" (UniqueName: \"kubernetes.io/projected/461d2f4d-16f9-41a1-835c-d0f726574833-kube-api-access-sfjj8\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:35.409517 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.409525    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/461d2f4d-16f9-41a1-835c-d0f726574833-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:35.409764 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.409534    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/461d2f4d-16f9-41a1-835c-d0f726574833-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:35.581088 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.581055    2557 generic.go:356] "Generic (PLEG): container finished" podID="461d2f4d-16f9-41a1-835c-d0f726574833" containerID="196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9" exitCode=0
Sep 05 02:48:35.581088 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.581082    2557 generic.go:356] "Generic (PLEG): container finished" podID="461d2f4d-16f9-41a1-835c-d0f726574833" containerID="ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a" exitCode=0
Sep 05 02:48:35.581307 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.581110    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"
Sep 05 02:48:35.581307 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.581146    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk" event={"ID":"461d2f4d-16f9-41a1-835c-d0f726574833","Type":"ContainerDied","Data":"196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9"}
Sep 05 02:48:35.581307 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.581212    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk" event={"ID":"461d2f4d-16f9-41a1-835c-d0f726574833","Type":"ContainerDied","Data":"ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a"}
Sep 05 02:48:35.581307 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.581245    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk" event={"ID":"461d2f4d-16f9-41a1-835c-d0f726574833","Type":"ContainerDied","Data":"9ac45f5be2b54f4c88289801a4c37c7baa7f0b6472627e33b88903e01d8c9960"}
Sep 05 02:48:35.581307 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.581276    2557 scope.go:117] "RemoveContainer" containerID="196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9"
Sep 05 02:48:35.591804 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.591780    2557 scope.go:117] "RemoveContainer" containerID="ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a"
Sep 05 02:48:35.599846 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.599822    2557 scope.go:117] "RemoveContainer" containerID="196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9"
Sep 05 02:48:35.600113 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:48:35.600090    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9\": container with ID starting with 196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9 not found: ID does not exist" containerID="196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9"
Sep 05 02:48:35.600180 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.600121    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9"} err="failed to get container status \"196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9\": rpc error: code = NotFound desc = could not find container \"196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9\": container with ID starting with 196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9 not found: ID does not exist"
Sep 05 02:48:35.600180 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.600142    2557 scope.go:117] "RemoveContainer" containerID="ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a"
Sep 05 02:48:35.600401 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:48:35.600381    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a\": container with ID starting with ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a not found: ID does not exist" containerID="ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a"
Sep 05 02:48:35.600461 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.600410    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a"} err="failed to get container status \"ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a\": rpc error: code = NotFound desc = could not find container \"ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a\": container with ID starting with ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a not found: ID does not exist"
Sep 05 02:48:35.600461 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.600431    2557 scope.go:117] "RemoveContainer" containerID="196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9"
Sep 05 02:48:35.600674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.600654    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9"} err="failed to get container status \"196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9\": rpc error: code = NotFound desc = could not find container \"196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9\": container with ID starting with 196d8d9900ebc377e6af4f1074f1a83761da8c4e98a009587c66c0b7e71c2eb9 not found: ID does not exist"
Sep 05 02:48:35.600728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.600675    2557 scope.go:117] "RemoveContainer" containerID="ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a"
Sep 05 02:48:35.600898 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.600875    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a"} err="failed to get container status \"ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a\": rpc error: code = NotFound desc = could not find container \"ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a\": container with ID starting with ad1b2b8371870a278ff681e73f331f5b299fe3f443401ff77e06171b3467f38a not found: ID does not exist"
Sep 05 02:48:35.604728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.604707    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"]
Sep 05 02:48:35.608996 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:35.608963    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-vlglk"]
Sep 05 02:48:36.991196 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:36.991153    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="461d2f4d-16f9-41a1-835c-d0f726574833" path="/var/lib/kubelet/pods/461d2f4d-16f9-41a1-835c-d0f726574833/volumes"
Sep 05 02:48:38.991605 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:38.991550    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:39.206564 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:39.206523    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:39.425966 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:39.425932    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"]
Sep 05 02:48:39.440111 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:39.440057    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:39.662064 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:39.662032    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:39.954125 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:39.954091    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"]
Sep 05 02:48:40.047774 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:40.047738    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:40.264214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:40.264114    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:41.398354 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:41.398314    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:42.404034 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.403998    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"]
Sep 05 02:48:42.404575 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.404552    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="461d2f4d-16f9-41a1-835c-d0f726574833" containerName="llm-d-routing-sidecar"
Sep 05 02:48:42.404674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.404576    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="461d2f4d-16f9-41a1-835c-d0f726574833" containerName="llm-d-routing-sidecar"
Sep 05 02:48:42.404674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.404599    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="461d2f4d-16f9-41a1-835c-d0f726574833" containerName="main"
Sep 05 02:48:42.404674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.404608    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="461d2f4d-16f9-41a1-835c-d0f726574833" containerName="main"
Sep 05 02:48:42.404674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.404622    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b86667b-243c-49ec-adc7-0bb1d5bc6b08" containerName="main"
Sep 05 02:48:42.404674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.404631    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b86667b-243c-49ec-adc7-0bb1d5bc6b08" containerName="main"
Sep 05 02:48:42.404926 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.404744    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="461d2f4d-16f9-41a1-835c-d0f726574833" containerName="llm-d-routing-sidecar"
Sep 05 02:48:42.404926 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.404760    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b86667b-243c-49ec-adc7-0bb1d5bc6b08" containerName="main"
Sep 05 02:48:42.404926 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.404768    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="461d2f4d-16f9-41a1-835c-d0f726574833" containerName="main"
Sep 05 02:48:42.409597 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.409563    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.411906 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.411880    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 05 02:48:42.412084 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.412066    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 05 02:48:42.412755 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.412732    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-kzrzb\""
Sep 05 02:48:42.412863 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.412768    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 05 02:48:42.418379 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.418352    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"]
Sep 05 02:48:42.443152 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.443122    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:42.476207 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.476179    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6krlg\" (UniqueName: \"kubernetes.io/projected/f134641c-e977-4b7c-a3e7-149e66aa3fed-kube-api-access-6krlg\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.476323 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.476217    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-model-cache\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.476323 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.476291    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-dshm\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.476400 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.476333    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.476400 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.476349    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f134641c-e977-4b7c-a3e7-149e66aa3fed-tls-certs\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.476400 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.476385    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-home\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.576968 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.576933    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-home\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.577130 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.577002    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krlg\" (UniqueName: \"kubernetes.io/projected/f134641c-e977-4b7c-a3e7-149e66aa3fed-kube-api-access-6krlg\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.577130 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.577034    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-model-cache\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.577130 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.577075    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-dshm\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.577130 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.577116    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.577359 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.577138    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f134641c-e977-4b7c-a3e7-149e66aa3fed-tls-certs\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.577359 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.577314    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-home\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.577535 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.577397    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-model-cache\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.577535 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.577466    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.579465 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.579422    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-dshm\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.579617 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.579598    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f134641c-e977-4b7c-a3e7-149e66aa3fed-tls-certs\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.584333 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.584314    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krlg\" (UniqueName: \"kubernetes.io/projected/f134641c-e977-4b7c-a3e7-149e66aa3fed-kube-api-access-6krlg\") pod \"test-div-v1-kserve-fb566d667-b756j\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.722507 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.722409    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:42.998010 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:42.997927    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"]
Sep 05 02:48:43.001223 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:48:43.001194    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf134641c_e977_4b7c_a3e7_149e66aa3fed.slice/crio-0d0f2b7d6cc5cf342a2b5db1939413c24bf7d955adb479c98355b40e1b8a4c88 WatchSource:0}: Error finding container 0d0f2b7d6cc5cf342a2b5db1939413c24bf7d955adb479c98355b40e1b8a4c88: Status 404 returned error can't find the container with id 0d0f2b7d6cc5cf342a2b5db1939413c24bf7d955adb479c98355b40e1b8a4c88
Sep 05 02:48:43.630495 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:43.630460    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j" event={"ID":"f134641c-e977-4b7c-a3e7-149e66aa3fed","Type":"ContainerStarted","Data":"65a44e06c03ac5a3203eb6a04e40fd237755fdc2b73ae0e4a45427e4f843b07f"}
Sep 05 02:48:43.630867 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:43.630515    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j" event={"ID":"f134641c-e977-4b7c-a3e7-149e66aa3fed","Type":"ContainerStarted","Data":"0d0f2b7d6cc5cf342a2b5db1939413c24bf7d955adb479c98355b40e1b8a4c88"}
Sep 05 02:48:43.648329 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:43.648273    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j" podStartSLOduration=1.648257718 podStartE2EDuration="1.648257718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:48:43.646422535 +0000 UTC m=+1827.201578844" watchObservedRunningTime="2026-09-05 02:48:43.648257718 +0000 UTC m=+1827.203414078"
Sep 05 02:48:43.722758 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:43.722727    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:44.014449 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:44.014317    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:47.008627 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:47.008593    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:47.108970 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:47.108931    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:48:52.723213 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:52.723178    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:52.736028 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:52.736005    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:53.695275 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:53.695240    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:48:54.447394 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.447346    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z" podUID="80ffc3f4-a7b9-452e-9381-937c8babac28" containerName="main" containerID="cri-o://83f9057a86303ada2df4a35ca4e7f34546a5d8b1922bd67dbeceee356126fbb9" gracePeriod=45
Sep 05 02:48:54.479299 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.479260    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z" podUID="80ffc3f4-a7b9-452e-9381-937c8babac28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 05 02:48:54.690762 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.690729    2557 generic.go:356] "Generic (PLEG): container finished" podID="80ffc3f4-a7b9-452e-9381-937c8babac28" containerID="83f9057a86303ada2df4a35ca4e7f34546a5d8b1922bd67dbeceee356126fbb9" exitCode=0
Sep 05 02:48:54.690907 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.690793    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z" event={"ID":"80ffc3f4-a7b9-452e-9381-937c8babac28","Type":"ContainerDied","Data":"83f9057a86303ada2df4a35ca4e7f34546a5d8b1922bd67dbeceee356126fbb9"}
Sep 05 02:48:54.690907 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.690880    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z" event={"ID":"80ffc3f4-a7b9-452e-9381-937c8babac28","Type":"ContainerDied","Data":"4f167090148ca3e65ab94101a15151f7f657ac8aa91097a8ce072a502feb9385"}
Sep 05 02:48:54.691037 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.690927    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f167090148ca3e65ab94101a15151f7f657ac8aa91097a8ce072a502feb9385"
Sep 05 02:48:54.703844 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.703785    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:54.781087 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.781054    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-model-cache\") pod \"80ffc3f4-a7b9-452e-9381-937c8babac28\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") "
Sep 05 02:48:54.781271 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.781103    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-home\") pod \"80ffc3f4-a7b9-452e-9381-937c8babac28\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") "
Sep 05 02:48:54.781271 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.781143    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2dgf\" (UniqueName: \"kubernetes.io/projected/80ffc3f4-a7b9-452e-9381-937c8babac28-kube-api-access-h2dgf\") pod \"80ffc3f4-a7b9-452e-9381-937c8babac28\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") "
Sep 05 02:48:54.781271 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.781167    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-tmp-dir\") pod \"80ffc3f4-a7b9-452e-9381-937c8babac28\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") "
Sep 05 02:48:54.781271 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.781184    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80ffc3f4-a7b9-452e-9381-937c8babac28-tls-certs\") pod \"80ffc3f4-a7b9-452e-9381-937c8babac28\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") "
Sep 05 02:48:54.781271 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.781230    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-dshm\") pod \"80ffc3f4-a7b9-452e-9381-937c8babac28\" (UID: \"80ffc3f4-a7b9-452e-9381-937c8babac28\") "
Sep 05 02:48:54.781569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.781409    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-model-cache" (OuterVolumeSpecName: "model-cache") pod "80ffc3f4-a7b9-452e-9381-937c8babac28" (UID: "80ffc3f4-a7b9-452e-9381-937c8babac28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:54.781569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.781544    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:54.781569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.781533    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80ffc3f4-a7b9-452e-9381-937c8babac28" (UID: "80ffc3f4-a7b9-452e-9381-937c8babac28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:54.781738 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.781585    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-home" (OuterVolumeSpecName: "home") pod "80ffc3f4-a7b9-452e-9381-937c8babac28" (UID: "80ffc3f4-a7b9-452e-9381-937c8babac28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:54.783529 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.783495    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-dshm" (OuterVolumeSpecName: "dshm") pod "80ffc3f4-a7b9-452e-9381-937c8babac28" (UID: "80ffc3f4-a7b9-452e-9381-937c8babac28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:54.783529 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.783500    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ffc3f4-a7b9-452e-9381-937c8babac28-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80ffc3f4-a7b9-452e-9381-937c8babac28" (UID: "80ffc3f4-a7b9-452e-9381-937c8babac28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:48:54.783677 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.783539    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ffc3f4-a7b9-452e-9381-937c8babac28-kube-api-access-h2dgf" (OuterVolumeSpecName: "kube-api-access-h2dgf") pod "80ffc3f4-a7b9-452e-9381-937c8babac28" (UID: "80ffc3f4-a7b9-452e-9381-937c8babac28"). InnerVolumeSpecName "kube-api-access-h2dgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:48:54.882661 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.882621    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:54.882661 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.882654    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:54.882912 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.882667    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2dgf\" (UniqueName: \"kubernetes.io/projected/80ffc3f4-a7b9-452e-9381-937c8babac28-kube-api-access-h2dgf\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:54.882912 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.882684    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80ffc3f4-a7b9-452e-9381-937c8babac28-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:54.882912 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.882695    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80ffc3f4-a7b9-452e-9381-937c8babac28-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:54.974293 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:54.974173    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8" podUID="6edd4dde-b03e-4b9f-ab4f-b91199400864" containerName="main" containerID="cri-o://75ab052fdd6093fca272f0b716391029b65fcad253cb35cf464ca7734d832a78" gracePeriod=45
Sep 05 02:48:55.222471 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.222428    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:55.286790 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.286705    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzwvj\" (UniqueName: \"kubernetes.io/projected/6edd4dde-b03e-4b9f-ab4f-b91199400864-kube-api-access-fzwvj\") pod \"6edd4dde-b03e-4b9f-ab4f-b91199400864\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") "
Sep 05 02:48:55.286790 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.286753    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-model-cache\") pod \"6edd4dde-b03e-4b9f-ab4f-b91199400864\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") "
Sep 05 02:48:55.287020 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.286886    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-dshm\") pod \"6edd4dde-b03e-4b9f-ab4f-b91199400864\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") "
Sep 05 02:48:55.287020 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.286935    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-home\") pod \"6edd4dde-b03e-4b9f-ab4f-b91199400864\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") "
Sep 05 02:48:55.287020 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.286974    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-tmp-dir\") pod \"6edd4dde-b03e-4b9f-ab4f-b91199400864\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") "
Sep 05 02:48:55.287188 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.287026    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-model-cache" (OuterVolumeSpecName: "model-cache") pod "6edd4dde-b03e-4b9f-ab4f-b91199400864" (UID: "6edd4dde-b03e-4b9f-ab4f-b91199400864"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:55.287188 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.287044    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edd4dde-b03e-4b9f-ab4f-b91199400864-tls-certs\") pod \"6edd4dde-b03e-4b9f-ab4f-b91199400864\" (UID: \"6edd4dde-b03e-4b9f-ab4f-b91199400864\") "
Sep 05 02:48:55.287286 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.287221    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-home" (OuterVolumeSpecName: "home") pod "6edd4dde-b03e-4b9f-ab4f-b91199400864" (UID: "6edd4dde-b03e-4b9f-ab4f-b91199400864"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:55.287286 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.287229    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6edd4dde-b03e-4b9f-ab4f-b91199400864" (UID: "6edd4dde-b03e-4b9f-ab4f-b91199400864"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:55.287386 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.287344    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:55.287386 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.287364    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:55.287386 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.287377    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:55.288877 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.288857    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edd4dde-b03e-4b9f-ab4f-b91199400864-kube-api-access-fzwvj" (OuterVolumeSpecName: "kube-api-access-fzwvj") pod "6edd4dde-b03e-4b9f-ab4f-b91199400864" (UID: "6edd4dde-b03e-4b9f-ab4f-b91199400864"). InnerVolumeSpecName "kube-api-access-fzwvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:48:55.289286 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.289269    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-dshm" (OuterVolumeSpecName: "dshm") pod "6edd4dde-b03e-4b9f-ab4f-b91199400864" (UID: "6edd4dde-b03e-4b9f-ab4f-b91199400864"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:48:55.289343 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.289294    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6edd4dde-b03e-4b9f-ab4f-b91199400864-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6edd4dde-b03e-4b9f-ab4f-b91199400864" (UID: "6edd4dde-b03e-4b9f-ab4f-b91199400864"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:48:55.388365 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.388329    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edd4dde-b03e-4b9f-ab4f-b91199400864-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:55.388365 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.388360    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edd4dde-b03e-4b9f-ab4f-b91199400864-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:55.388365 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.388371    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzwvj\" (UniqueName: \"kubernetes.io/projected/6edd4dde-b03e-4b9f-ab4f-b91199400864-kube-api-access-fzwvj\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:48:55.697656 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.697617    2557 generic.go:356] "Generic (PLEG): container finished" podID="6edd4dde-b03e-4b9f-ab4f-b91199400864" containerID="75ab052fdd6093fca272f0b716391029b65fcad253cb35cf464ca7734d832a78" exitCode=0
Sep 05 02:48:55.698163 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.697661    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"
Sep 05 02:48:55.698163 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.697699    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8" event={"ID":"6edd4dde-b03e-4b9f-ab4f-b91199400864","Type":"ContainerDied","Data":"75ab052fdd6093fca272f0b716391029b65fcad253cb35cf464ca7734d832a78"}
Sep 05 02:48:55.698163 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.697745    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8" event={"ID":"6edd4dde-b03e-4b9f-ab4f-b91199400864","Type":"ContainerDied","Data":"e7873ba9504c3a476fa1823a5389ad9e5bd8dab4b07aa07fa75bce6ffbfb6b92"}
Sep 05 02:48:55.698163 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.697767    2557 scope.go:117] "RemoveContainer" containerID="75ab052fdd6093fca272f0b716391029b65fcad253cb35cf464ca7734d832a78"
Sep 05 02:48:55.698163 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.697889    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"
Sep 05 02:48:55.707030 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.707006    2557 scope.go:117] "RemoveContainer" containerID="75ab052fdd6093fca272f0b716391029b65fcad253cb35cf464ca7734d832a78"
Sep 05 02:48:55.707336 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:48:55.707314    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ab052fdd6093fca272f0b716391029b65fcad253cb35cf464ca7734d832a78\": container with ID starting with 75ab052fdd6093fca272f0b716391029b65fcad253cb35cf464ca7734d832a78 not found: ID does not exist" containerID="75ab052fdd6093fca272f0b716391029b65fcad253cb35cf464ca7734d832a78"
Sep 05 02:48:55.707417 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.707344    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ab052fdd6093fca272f0b716391029b65fcad253cb35cf464ca7734d832a78"} err="failed to get container status \"75ab052fdd6093fca272f0b716391029b65fcad253cb35cf464ca7734d832a78\": rpc error: code = NotFound desc = could not find container \"75ab052fdd6093fca272f0b716391029b65fcad253cb35cf464ca7734d832a78\": container with ID starting with 75ab052fdd6093fca272f0b716391029b65fcad253cb35cf464ca7734d832a78 not found: ID does not exist"
Sep 05 02:48:55.714066 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.714040    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"]
Sep 05 02:48:55.717818 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.717792    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-zfc5z"]
Sep 05 02:48:55.727387 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.727363    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"]
Sep 05 02:48:55.731011 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:55.730986    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-868j8"]
Sep 05 02:48:56.991128 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:56.991094    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6edd4dde-b03e-4b9f-ab4f-b91199400864" path="/var/lib/kubelet/pods/6edd4dde-b03e-4b9f-ab4f-b91199400864/volumes"
Sep 05 02:48:56.991538 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:48:56.991522    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80ffc3f4-a7b9-452e-9381-937c8babac28" path="/var/lib/kubelet/pods/80ffc3f4-a7b9-452e-9381-937c8babac28/volumes"
Sep 05 02:49:11.205604 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.205563    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"]
Sep 05 02:49:11.206068 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.205932    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ffc3f4-a7b9-452e-9381-937c8babac28" containerName="main"
Sep 05 02:49:11.206068 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.205943    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ffc3f4-a7b9-452e-9381-937c8babac28" containerName="main"
Sep 05 02:49:11.206068 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.205958    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6edd4dde-b03e-4b9f-ab4f-b91199400864" containerName="main"
Sep 05 02:49:11.206068 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.205964    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6edd4dde-b03e-4b9f-ab4f-b91199400864" containerName="main"
Sep 05 02:49:11.206068 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.206017    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6edd4dde-b03e-4b9f-ab4f-b91199400864" containerName="main"
Sep 05 02:49:11.206068 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.206028    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ffc3f4-a7b9-452e-9381-937c8babac28" containerName="main"
Sep 05 02:49:11.210359 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.210341    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.212579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.212555    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 05 02:49:11.218990 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.218967    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"]
Sep 05 02:49:11.301299 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.301257    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:11.324862 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.324823    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d56c96-d67a-49d3-9121-876a891093bc-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.325033 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.324874    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.325033 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.324935    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-home\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.325033 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.324951    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.325033 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.324999    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.325172 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.325034    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vjcq\" (UniqueName: \"kubernetes.io/projected/41d56c96-d67a-49d3-9121-876a891093bc-kube-api-access-7vjcq\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.425753 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.425702    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.425753 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.425765    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vjcq\" (UniqueName: \"kubernetes.io/projected/41d56c96-d67a-49d3-9121-876a891093bc-kube-api-access-7vjcq\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.426000 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.425808    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d56c96-d67a-49d3-9121-876a891093bc-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.426000 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.425834    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.426000 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.425863    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-home\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.426000 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.425885    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.426213 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.426189    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.426291 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.426232    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.426355 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.426325    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-home\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.428196 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.428165    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.428327 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.428308    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d56c96-d67a-49d3-9121-876a891093bc-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.433952 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.433924    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vjcq\" (UniqueName: \"kubernetes.io/projected/41d56c96-d67a-49d3-9121-876a891093bc-kube-api-access-7vjcq\") pod \"test-div-v2-kserve-5f6fdb6b77-dw8qf\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.521317 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.521218    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:11.817671 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:11.817639    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"]
Sep 05 02:49:11.820025 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:49:11.819994    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d56c96_d67a_49d3_9121_876a891093bc.slice/crio-ff90402df39801dda230ca0b96d2d94dade5e5165ea3dad4729e9bf5a867b0d4 WatchSource:0}: Error finding container ff90402df39801dda230ca0b96d2d94dade5e5165ea3dad4729e9bf5a867b0d4: Status 404 returned error can't find the container with id ff90402df39801dda230ca0b96d2d94dade5e5165ea3dad4729e9bf5a867b0d4
Sep 05 02:49:12.792807 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:12.792763    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf" event={"ID":"41d56c96-d67a-49d3-9121-876a891093bc","Type":"ContainerStarted","Data":"5b982c8c4d12fd0cb991604a2909aa4866391237fc73762df64f1d2a6d4ac321"}
Sep 05 02:49:12.793274 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:12.792860    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf" event={"ID":"41d56c96-d67a-49d3-9121-876a891093bc","Type":"ContainerStarted","Data":"ff90402df39801dda230ca0b96d2d94dade5e5165ea3dad4729e9bf5a867b0d4"}
Sep 05 02:49:12.814594 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:12.814538    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf" podStartSLOduration=1.8145179580000002 podStartE2EDuration="1.814517958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:49:12.811644052 +0000 UTC m=+1856.366800362" watchObservedRunningTime="2026-09-05 02:49:12.814517958 +0000 UTC m=+1856.369674267"
Sep 05 02:49:13.521483 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:13.521413    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:21.431824 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:21.431783    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:21.521467 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:21.521422    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:21.533931 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:21.533909    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:21.851214 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:21.851183    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:49:25.522825 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:25.522777    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:25.622161 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:25.622129    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:27.029696 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:27.029663    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:27.174678 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:27.174642    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:30.786268 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:30.786232    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:34.847423 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:34.847389    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"]
Sep 05 02:49:34.850755 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:34.850733    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:34.853077 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:34.853048    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 05 02:49:34.853203 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:34.853101    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 05 02:49:34.853978 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:34.853960    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-brvv8\""
Sep 05 02:49:34.854088 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:34.854021    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 05 02:49:34.860978 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:34.860953    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"]
Sep 05 02:49:34.875885 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:34.875861    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:34.940985 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:34.940947    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbkzk\" (UniqueName: \"kubernetes.io/projected/f517b39f-bcdb-45d3-b51d-f3e84f999973-kube-api-access-gbkzk\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:34.941141 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:34.940997    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:34.941141 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:34.941029    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:34.941141 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:34.941046    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f517b39f-bcdb-45d3-b51d-f3e84f999973-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:34.941141 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:34.941065    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:34.941141 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:34.941113    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:35.042310 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.042279    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:35.042524 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.042339    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbkzk\" (UniqueName: \"kubernetes.io/projected/f517b39f-bcdb-45d3-b51d-f3e84f999973-kube-api-access-gbkzk\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:35.042524 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.042367    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:35.042524 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.042398    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:35.042524 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.042416    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f517b39f-bcdb-45d3-b51d-f3e84f999973-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:35.042524 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.042446    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:35.042870 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.042838    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:35.042870 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.042850    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:35.043029 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.042865    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:35.044561 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.044531    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:35.044806 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.044787    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f517b39f-bcdb-45d3-b51d-f3e84f999973-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:35.050891 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.050870    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbkzk\" (UniqueName: \"kubernetes.io/projected/f517b39f-bcdb-45d3-b51d-f3e84f999973-kube-api-access-gbkzk\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:35.162634 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.162551    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:35.228464 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.228402    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:35.441456 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.441407    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"]
Sep 05 02:49:35.443349 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:49:35.443322    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf517b39f_bcdb_45d3_b51d_f3e84f999973.slice/crio-3261614ae55b212815abc68fce248fb08d4e2cdb0804eabbe359bfdba9eb34f2 WatchSource:0}: Error finding container 3261614ae55b212815abc68fce248fb08d4e2cdb0804eabbe359bfdba9eb34f2: Status 404 returned error can't find the container with id 3261614ae55b212815abc68fce248fb08d4e2cdb0804eabbe359bfdba9eb34f2
Sep 05 02:49:35.915900 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.915865    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx" event={"ID":"f517b39f-bcdb-45d3-b51d-f3e84f999973","Type":"ContainerStarted","Data":"8921015dc52c99467133bb76296f4a42f5e2fd4de2807885e9f0f6abad37a89b"}
Sep 05 02:49:35.916271 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.915919    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx" event={"ID":"f517b39f-bcdb-45d3-b51d-f3e84f999973","Type":"ContainerStarted","Data":"3261614ae55b212815abc68fce248fb08d4e2cdb0804eabbe359bfdba9eb34f2"}
Sep 05 02:49:35.941763 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:35.941698    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx" podStartSLOduration=1.941678435 podStartE2EDuration="1.941678435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:49:35.938250056 +0000 UTC m=+1879.493406366" watchObservedRunningTime="2026-09-05 02:49:35.941678435 +0000 UTC m=+1879.496834745"
Sep 05 02:49:36.163690 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:36.163650    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:36.777270 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:36.777224    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:36.877691 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:36.877646    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:40.797459 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:40.797348    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:40.897899 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:40.897859    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:45.163652 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:45.163616    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:45.176484 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:45.176453    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:45.985519 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:45.985482    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:49:53.699676 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:53.699632    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:56.345901 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:56.345856    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:56.446983 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:56.446948    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:56.553086 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:56.553051    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:57.124849 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:57.124808    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:57.260048 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:57.260012    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:49:58.223566 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:49:58.223527    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:00.515384 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:00.515335    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"]
Sep 05 02:50:00.529179 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:00.529154    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:01.037665 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:01.037613    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"]
Sep 05 02:50:01.060489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:01.060457    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:01.420178 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:01.420135    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:03.237304 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:03.237266    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:08.824510 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:08.824470    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:08.896281 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:08.896240    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:13.303510 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.303469    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"]
Sep 05 02:50:13.307205 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.307168    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.310591 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.310558    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 05 02:50:13.310591 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.310576    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 05 02:50:13.310766 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.310591    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-hd4xj\""
Sep 05 02:50:13.310766 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.310615    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 05 02:50:13.319224 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.319189    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"]
Sep 05 02:50:13.343576 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.343543    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:13.399911 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.399876    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.400055 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.399936    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-home\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.400055 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.399972    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-dshm\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.400055 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.399996    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-model-cache\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.400055 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.400039    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56c84102-ed66-4be6-b289-639ad5c62c83-tls-certs\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.400243 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.400087    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq6q4\" (UniqueName: \"kubernetes.io/projected/56c84102-ed66-4be6-b289-639ad5c62c83-kube-api-access-wq6q4\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.500672 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.500633    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-home\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.500862 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.500683    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-dshm\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.500862 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.500719    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-model-cache\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.500862 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.500739    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56c84102-ed66-4be6-b289-639ad5c62c83-tls-certs\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.500862 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.500760    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq6q4\" (UniqueName: \"kubernetes.io/projected/56c84102-ed66-4be6-b289-639ad5c62c83-kube-api-access-wq6q4\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.500862 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.500817    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.501143 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.501108    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-home\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.501203 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.501157    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-model-cache\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.501259 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.501235    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.503173 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.503140    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-dshm\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.503416 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.503400    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56c84102-ed66-4be6-b289-639ad5c62c83-tls-certs\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.508546 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.508524    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq6q4\" (UniqueName: \"kubernetes.io/projected/56c84102-ed66-4be6-b289-639ad5c62c83-kube-api-access-wq6q4\") pod \"stop-v1-kserve-5657b88c95-9vz5v\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.620804 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.620753    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:13.898416 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:13.898387    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"]
Sep 05 02:50:13.900241 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:50:13.900206    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c84102_ed66_4be6_b289_639ad5c62c83.slice/crio-245f02df2497e674eb37bb05dcad4d762290467c3ff1025e0950faf1f7bb8f22 WatchSource:0}: Error finding container 245f02df2497e674eb37bb05dcad4d762290467c3ff1025e0950faf1f7bb8f22: Status 404 returned error can't find the container with id 245f02df2497e674eb37bb05dcad4d762290467c3ff1025e0950faf1f7bb8f22
Sep 05 02:50:14.126900 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:14.126858    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v" event={"ID":"56c84102-ed66-4be6-b289-639ad5c62c83","Type":"ContainerStarted","Data":"701e6105feef2d3d2a113d597337d6979edfe0f9406e7a35888001c68d1b6233"}
Sep 05 02:50:14.127069 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:14.126951    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v" event={"ID":"56c84102-ed66-4be6-b289-639ad5c62c83","Type":"ContainerStarted","Data":"245f02df2497e674eb37bb05dcad4d762290467c3ff1025e0950faf1f7bb8f22"}
Sep 05 02:50:14.148082 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:14.147992    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v" podStartSLOduration=1.147976988 podStartE2EDuration="1.147976988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:50:14.145880142 +0000 UTC m=+1917.701036453" watchObservedRunningTime="2026-09-05 02:50:14.147976988 +0000 UTC m=+1917.703133297"
Sep 05 02:50:14.621053 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:14.621016    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:15.534896 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.534837    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf" podUID="41d56c96-d67a-49d3-9121-876a891093bc" containerName="main" containerID="cri-o://5b982c8c4d12fd0cb991604a2909aa4866391237fc73762df64f1d2a6d4ac321" gracePeriod=45
Sep 05 02:50:15.783504 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.783478    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:50:15.821174 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.821149    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d56c96-d67a-49d3-9121-876a891093bc-tls-certs\") pod \"41d56c96-d67a-49d3-9121-876a891093bc\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") "
Sep 05 02:50:15.821376 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.821253    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-model-cache\") pod \"41d56c96-d67a-49d3-9121-876a891093bc\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") "
Sep 05 02:50:15.821376 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.821299    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-tmp-dir\") pod \"41d56c96-d67a-49d3-9121-876a891093bc\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") "
Sep 05 02:50:15.821376 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.821315    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vjcq\" (UniqueName: \"kubernetes.io/projected/41d56c96-d67a-49d3-9121-876a891093bc-kube-api-access-7vjcq\") pod \"41d56c96-d67a-49d3-9121-876a891093bc\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") "
Sep 05 02:50:15.821376 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.821332    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-home\") pod \"41d56c96-d67a-49d3-9121-876a891093bc\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") "
Sep 05 02:50:15.821376 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.821349    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-dshm\") pod \"41d56c96-d67a-49d3-9121-876a891093bc\" (UID: \"41d56c96-d67a-49d3-9121-876a891093bc\") "
Sep 05 02:50:15.821671 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.821588    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "41d56c96-d67a-49d3-9121-876a891093bc" (UID: "41d56c96-d67a-49d3-9121-876a891093bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:15.821671 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.821608    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41d56c96-d67a-49d3-9121-876a891093bc" (UID: "41d56c96-d67a-49d3-9121-876a891093bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:15.821947 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.821922    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-home" (OuterVolumeSpecName: "home") pod "41d56c96-d67a-49d3-9121-876a891093bc" (UID: "41d56c96-d67a-49d3-9121-876a891093bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:15.823541 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.823511    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d56c96-d67a-49d3-9121-876a891093bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41d56c96-d67a-49d3-9121-876a891093bc" (UID: "41d56c96-d67a-49d3-9121-876a891093bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:50:15.823694 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.823675    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d56c96-d67a-49d3-9121-876a891093bc-kube-api-access-7vjcq" (OuterVolumeSpecName: "kube-api-access-7vjcq") pod "41d56c96-d67a-49d3-9121-876a891093bc" (UID: "41d56c96-d67a-49d3-9121-876a891093bc"). InnerVolumeSpecName "kube-api-access-7vjcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:50:15.823814 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.823797    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-dshm" (OuterVolumeSpecName: "dshm") pod "41d56c96-d67a-49d3-9121-876a891093bc" (UID: "41d56c96-d67a-49d3-9121-876a891093bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:15.922602 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.922561    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:15.922602 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.922597    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:15.922602 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.922607    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vjcq\" (UniqueName: \"kubernetes.io/projected/41d56c96-d67a-49d3-9121-876a891093bc-kube-api-access-7vjcq\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:15.922835 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.922618    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:15.922835 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.922627    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d56c96-d67a-49d3-9121-876a891093bc-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:15.922835 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:15.922635    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d56c96-d67a-49d3-9121-876a891093bc-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:16.057624 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.057580    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j" podUID="f134641c-e977-4b7c-a3e7-149e66aa3fed" containerName="main" containerID="cri-o://65a44e06c03ac5a3203eb6a04e40fd237755fdc2b73ae0e4a45427e4f843b07f" gracePeriod=45
Sep 05 02:50:16.140215 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.140175    2557 generic.go:356] "Generic (PLEG): container finished" podID="41d56c96-d67a-49d3-9121-876a891093bc" containerID="5b982c8c4d12fd0cb991604a2909aa4866391237fc73762df64f1d2a6d4ac321" exitCode=0
Sep 05 02:50:16.140408 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.140230    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"
Sep 05 02:50:16.140408 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.140239    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf" event={"ID":"41d56c96-d67a-49d3-9121-876a891093bc","Type":"ContainerDied","Data":"5b982c8c4d12fd0cb991604a2909aa4866391237fc73762df64f1d2a6d4ac321"}
Sep 05 02:50:16.140408 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.140301    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf" event={"ID":"41d56c96-d67a-49d3-9121-876a891093bc","Type":"ContainerDied","Data":"ff90402df39801dda230ca0b96d2d94dade5e5165ea3dad4729e9bf5a867b0d4"}
Sep 05 02:50:16.140408 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.140326    2557 scope.go:117] "RemoveContainer" containerID="5b982c8c4d12fd0cb991604a2909aa4866391237fc73762df64f1d2a6d4ac321"
Sep 05 02:50:16.162565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.162536    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"]
Sep 05 02:50:16.166396 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.166368    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-dw8qf"]
Sep 05 02:50:16.170011 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.169983    2557 scope.go:117] "RemoveContainer" containerID="5b982c8c4d12fd0cb991604a2909aa4866391237fc73762df64f1d2a6d4ac321"
Sep 05 02:50:16.170368 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:50:16.170343    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b982c8c4d12fd0cb991604a2909aa4866391237fc73762df64f1d2a6d4ac321\": container with ID starting with 5b982c8c4d12fd0cb991604a2909aa4866391237fc73762df64f1d2a6d4ac321 not found: ID does not exist" containerID="5b982c8c4d12fd0cb991604a2909aa4866391237fc73762df64f1d2a6d4ac321"
Sep 05 02:50:16.170460 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.170378    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b982c8c4d12fd0cb991604a2909aa4866391237fc73762df64f1d2a6d4ac321"} err="failed to get container status \"5b982c8c4d12fd0cb991604a2909aa4866391237fc73762df64f1d2a6d4ac321\": rpc error: code = NotFound desc = could not find container \"5b982c8c4d12fd0cb991604a2909aa4866391237fc73762df64f1d2a6d4ac321\": container with ID starting with 5b982c8c4d12fd0cb991604a2909aa4866391237fc73762df64f1d2a6d4ac321 not found: ID does not exist"
Sep 05 02:50:16.307901 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.307873    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:50:16.326119 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.325987    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-tmp-dir\") pod \"f134641c-e977-4b7c-a3e7-149e66aa3fed\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") "
Sep 05 02:50:16.326119 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.326071    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6krlg\" (UniqueName: \"kubernetes.io/projected/f134641c-e977-4b7c-a3e7-149e66aa3fed-kube-api-access-6krlg\") pod \"f134641c-e977-4b7c-a3e7-149e66aa3fed\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") "
Sep 05 02:50:16.326286 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.326135    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-dshm\") pod \"f134641c-e977-4b7c-a3e7-149e66aa3fed\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") "
Sep 05 02:50:16.326286 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.326166    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-model-cache\") pod \"f134641c-e977-4b7c-a3e7-149e66aa3fed\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") "
Sep 05 02:50:16.326286 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.326220    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f134641c-e977-4b7c-a3e7-149e66aa3fed-tls-certs\") pod \"f134641c-e977-4b7c-a3e7-149e66aa3fed\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") "
Sep 05 02:50:16.326286 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.326244    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-home\") pod \"f134641c-e977-4b7c-a3e7-149e66aa3fed\" (UID: \"f134641c-e977-4b7c-a3e7-149e66aa3fed\") "
Sep 05 02:50:16.327646 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.326572    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f134641c-e977-4b7c-a3e7-149e66aa3fed" (UID: "f134641c-e977-4b7c-a3e7-149e66aa3fed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:16.327646 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.326813    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-home" (OuterVolumeSpecName: "home") pod "f134641c-e977-4b7c-a3e7-149e66aa3fed" (UID: "f134641c-e977-4b7c-a3e7-149e66aa3fed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:16.327646 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.327632    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-model-cache" (OuterVolumeSpecName: "model-cache") pod "f134641c-e977-4b7c-a3e7-149e66aa3fed" (UID: "f134641c-e977-4b7c-a3e7-149e66aa3fed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:16.331030 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.330897    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-dshm" (OuterVolumeSpecName: "dshm") pod "f134641c-e977-4b7c-a3e7-149e66aa3fed" (UID: "f134641c-e977-4b7c-a3e7-149e66aa3fed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:16.331030 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.330971    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f134641c-e977-4b7c-a3e7-149e66aa3fed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f134641c-e977-4b7c-a3e7-149e66aa3fed" (UID: "f134641c-e977-4b7c-a3e7-149e66aa3fed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:50:16.331511 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.331483    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f134641c-e977-4b7c-a3e7-149e66aa3fed-kube-api-access-6krlg" (OuterVolumeSpecName: "kube-api-access-6krlg") pod "f134641c-e977-4b7c-a3e7-149e66aa3fed" (UID: "f134641c-e977-4b7c-a3e7-149e66aa3fed"). InnerVolumeSpecName "kube-api-access-6krlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:50:16.427433 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.427390    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f134641c-e977-4b7c-a3e7-149e66aa3fed-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:16.427433 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.427429    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:16.427653 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.427465    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:16.427653 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.427479    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6krlg\" (UniqueName: \"kubernetes.io/projected/f134641c-e977-4b7c-a3e7-149e66aa3fed-kube-api-access-6krlg\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:16.427653 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.427493    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:16.427653 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.427506    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f134641c-e977-4b7c-a3e7-149e66aa3fed-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:16.992854 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:16.992821    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41d56c96-d67a-49d3-9121-876a891093bc" path="/var/lib/kubelet/pods/41d56c96-d67a-49d3-9121-876a891093bc/volumes"
Sep 05 02:50:17.148987 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:17.148954    2557 generic.go:356] "Generic (PLEG): container finished" podID="f134641c-e977-4b7c-a3e7-149e66aa3fed" containerID="65a44e06c03ac5a3203eb6a04e40fd237755fdc2b73ae0e4a45427e4f843b07f" exitCode=0
Sep 05 02:50:17.149160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:17.148997    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"
Sep 05 02:50:17.149160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:17.149038    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j" event={"ID":"f134641c-e977-4b7c-a3e7-149e66aa3fed","Type":"ContainerDied","Data":"65a44e06c03ac5a3203eb6a04e40fd237755fdc2b73ae0e4a45427e4f843b07f"}
Sep 05 02:50:17.149160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:17.149099    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j" event={"ID":"f134641c-e977-4b7c-a3e7-149e66aa3fed","Type":"ContainerDied","Data":"0d0f2b7d6cc5cf342a2b5db1939413c24bf7d955adb479c98355b40e1b8a4c88"}
Sep 05 02:50:17.149160 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:17.149129    2557 scope.go:117] "RemoveContainer" containerID="65a44e06c03ac5a3203eb6a04e40fd237755fdc2b73ae0e4a45427e4f843b07f"
Sep 05 02:50:17.158346 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:17.158327    2557 scope.go:117] "RemoveContainer" containerID="65a44e06c03ac5a3203eb6a04e40fd237755fdc2b73ae0e4a45427e4f843b07f"
Sep 05 02:50:17.158635 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:50:17.158612    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a44e06c03ac5a3203eb6a04e40fd237755fdc2b73ae0e4a45427e4f843b07f\": container with ID starting with 65a44e06c03ac5a3203eb6a04e40fd237755fdc2b73ae0e4a45427e4f843b07f not found: ID does not exist" containerID="65a44e06c03ac5a3203eb6a04e40fd237755fdc2b73ae0e4a45427e4f843b07f"
Sep 05 02:50:17.158696 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:17.158643    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a44e06c03ac5a3203eb6a04e40fd237755fdc2b73ae0e4a45427e4f843b07f"} err="failed to get container status \"65a44e06c03ac5a3203eb6a04e40fd237755fdc2b73ae0e4a45427e4f843b07f\": rpc error: code = NotFound desc = could not find container \"65a44e06c03ac5a3203eb6a04e40fd237755fdc2b73ae0e4a45427e4f843b07f\": container with ID starting with 65a44e06c03ac5a3203eb6a04e40fd237755fdc2b73ae0e4a45427e4f843b07f not found: ID does not exist"
Sep 05 02:50:17.165263 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:17.165240    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"]
Sep 05 02:50:17.168853 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:17.168829    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-b756j"]
Sep 05 02:50:18.892281 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.892235    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"]
Sep 05 02:50:18.892846 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.892821    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d56c96-d67a-49d3-9121-876a891093bc" containerName="main"
Sep 05 02:50:18.892846 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.892844    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d56c96-d67a-49d3-9121-876a891093bc" containerName="main"
Sep 05 02:50:18.893054 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.892863    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f134641c-e977-4b7c-a3e7-149e66aa3fed" containerName="main"
Sep 05 02:50:18.893054 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.892870    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f134641c-e977-4b7c-a3e7-149e66aa3fed" containerName="main"
Sep 05 02:50:18.893054 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.892940    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f134641c-e977-4b7c-a3e7-149e66aa3fed" containerName="main"
Sep 05 02:50:18.893054 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.892950    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d56c96-d67a-49d3-9121-876a891093bc" containerName="main"
Sep 05 02:50:18.896908 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.896889    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:18.899240 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.899212    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 05 02:50:18.907427 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.907402    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"]
Sep 05 02:50:18.928231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.928203    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:18.946431 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.946407    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:18.946584 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.946472    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-model-cache\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:18.946584 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.946511    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z55fp\" (UniqueName: \"kubernetes.io/projected/b825918b-5c03-4dc5-afe8-e5dc755509f7-kube-api-access-z55fp\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:18.946584 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.946544    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-home\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:18.946700 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.946594    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-dshm\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:18.946700 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.946616    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b825918b-5c03-4dc5-afe8-e5dc755509f7-tls-certs\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:18.991523 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:18.991492    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f134641c-e977-4b7c-a3e7-149e66aa3fed" path="/var/lib/kubelet/pods/f134641c-e977-4b7c-a3e7-149e66aa3fed/volumes"
Sep 05 02:50:19.047743 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:19.047707    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:19.047743 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:19.047749    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-model-cache\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:19.047966 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:19.047776    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z55fp\" (UniqueName: \"kubernetes.io/projected/b825918b-5c03-4dc5-afe8-e5dc755509f7-kube-api-access-z55fp\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:19.047966 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:19.047804    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-home\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:19.047966 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:19.047828    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-dshm\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:19.048124 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:19.048009    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b825918b-5c03-4dc5-afe8-e5dc755509f7-tls-certs\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:19.048183 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:19.048137    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-model-cache\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:19.048235 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:19.048196    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:19.048299 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:19.048281    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-home\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:19.050050 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:19.050022    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-dshm\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:19.050481 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:19.050454    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b825918b-5c03-4dc5-afe8-e5dc755509f7-tls-certs\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:19.055237 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:19.055216    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z55fp\" (UniqueName: \"kubernetes.io/projected/b825918b-5c03-4dc5-afe8-e5dc755509f7-kube-api-access-z55fp\") pod \"stop-v2-kserve-7f68747c45-lx5sj\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:19.208681 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:19.208590    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:19.491126 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:19.491097    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"]
Sep 05 02:50:19.492951 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:50:19.492920    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb825918b_5c03_4dc5_afe8_e5dc755509f7.slice/crio-a2585ec885a387629731296a44bdf1b19d9fcb6c7c65a44ad9d323fb4a43540e WatchSource:0}: Error finding container a2585ec885a387629731296a44bdf1b19d9fcb6c7c65a44ad9d323fb4a43540e: Status 404 returned error can't find the container with id a2585ec885a387629731296a44bdf1b19d9fcb6c7c65a44ad9d323fb4a43540e
Sep 05 02:50:20.169469 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:20.169409    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj" event={"ID":"b825918b-5c03-4dc5-afe8-e5dc755509f7","Type":"ContainerStarted","Data":"49148f36676bde811a15629dd9bbabb53b512619977e446dcdb5b9de5ad052c5"}
Sep 05 02:50:20.169856 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:20.169491    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj" event={"ID":"b825918b-5c03-4dc5-afe8-e5dc755509f7","Type":"ContainerStarted","Data":"a2585ec885a387629731296a44bdf1b19d9fcb6c7c65a44ad9d323fb4a43540e"}
Sep 05 02:50:20.193768 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:20.193716    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj" podStartSLOduration=2.193700758 podStartE2EDuration="2.193700758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:50:20.191888812 +0000 UTC m=+1923.747045122" watchObservedRunningTime="2026-09-05 02:50:20.193700758 +0000 UTC m=+1923.748857067"
Sep 05 02:50:20.209633 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:20.209605    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:23.621255 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:23.621214    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:23.635151 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:23.635122    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:24.208313 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:24.208278    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:50:29.209502 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:29.209471    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:29.222027 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:29.222000    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:29.235278 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:29.235256    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:50:31.270310 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:31.270276    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:36.320554 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:36.320516    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:36.387552 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:36.387511    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:40.952547 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:40.952506    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"]
Sep 05 02:50:40.953934 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:40.953898    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:40.962479 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:50:40.962415    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:50:40.962479 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:50:40.962479 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:50:40.962479 ip-10-0-143-128 kubenswrapper[2557]:  > containerID="8921015dc52c99467133bb76296f4a42f5e2fd4de2807885e9f0f6abad37a89b" cmd=["/bin/sleep","15"]
Sep 05 02:50:40.962777 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:40.962489    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx" message="" err=<
Sep 05 02:50:40.962777 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:50:40.962777 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:50:40.962777 ip-10-0-143-128 kubenswrapper[2557]:  >
Sep 05 02:50:40.962777 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:50:40.962503    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:50:40.962777 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:50:40.962777 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:50:40.962777 ip-10-0-143-128 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx" podUID="f517b39f-bcdb-45d3-b51d-f3e84f999973" containerName="main" containerID="cri-o://8921015dc52c99467133bb76296f4a42f5e2fd4de2807885e9f0f6abad37a89b"
Sep 05 02:50:40.962777 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:40.962550    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx" podUID="f517b39f-bcdb-45d3-b51d-f3e84f999973" containerName="main" containerID="cri-o://8921015dc52c99467133bb76296f4a42f5e2fd4de2807885e9f0f6abad37a89b" gracePeriod=60
Sep 05 02:50:40.974961 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:40.974933    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx" podUID="f517b39f-bcdb-45d3-b51d-f3e84f999973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 05 02:50:41.090855 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.090817    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:50:41.292963 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.292876    2557 generic.go:356] "Generic (PLEG): container finished" podID="f517b39f-bcdb-45d3-b51d-f3e84f999973" containerID="8921015dc52c99467133bb76296f4a42f5e2fd4de2807885e9f0f6abad37a89b" exitCode=0
Sep 05 02:50:41.292963 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.292948    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx" event={"ID":"f517b39f-bcdb-45d3-b51d-f3e84f999973","Type":"ContainerDied","Data":"8921015dc52c99467133bb76296f4a42f5e2fd4de2807885e9f0f6abad37a89b"}
Sep 05 02:50:41.721520 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.721494    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:50:41.866787 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.866691    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f517b39f-bcdb-45d3-b51d-f3e84f999973-tls-certs\") pod \"f517b39f-bcdb-45d3-b51d-f3e84f999973\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") "
Sep 05 02:50:41.866787 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.866771    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-tmp-dir\") pod \"f517b39f-bcdb-45d3-b51d-f3e84f999973\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") "
Sep 05 02:50:41.867018 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.866803    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbkzk\" (UniqueName: \"kubernetes.io/projected/f517b39f-bcdb-45d3-b51d-f3e84f999973-kube-api-access-gbkzk\") pod \"f517b39f-bcdb-45d3-b51d-f3e84f999973\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") "
Sep 05 02:50:41.867018 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.866842    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-model-cache\") pod \"f517b39f-bcdb-45d3-b51d-f3e84f999973\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") "
Sep 05 02:50:41.867018 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.866870    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-home\") pod \"f517b39f-bcdb-45d3-b51d-f3e84f999973\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") "
Sep 05 02:50:41.867018 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.866890    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-dshm\") pod \"f517b39f-bcdb-45d3-b51d-f3e84f999973\" (UID: \"f517b39f-bcdb-45d3-b51d-f3e84f999973\") "
Sep 05 02:50:41.867243 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.867041    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f517b39f-bcdb-45d3-b51d-f3e84f999973" (UID: "f517b39f-bcdb-45d3-b51d-f3e84f999973"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:41.867303 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.867249    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:41.867355 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.867311    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-home" (OuterVolumeSpecName: "home") pod "f517b39f-bcdb-45d3-b51d-f3e84f999973" (UID: "f517b39f-bcdb-45d3-b51d-f3e84f999973"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:41.867413 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.867380    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-model-cache" (OuterVolumeSpecName: "model-cache") pod "f517b39f-bcdb-45d3-b51d-f3e84f999973" (UID: "f517b39f-bcdb-45d3-b51d-f3e84f999973"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:41.869339 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.869310    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f517b39f-bcdb-45d3-b51d-f3e84f999973-kube-api-access-gbkzk" (OuterVolumeSpecName: "kube-api-access-gbkzk") pod "f517b39f-bcdb-45d3-b51d-f3e84f999973" (UID: "f517b39f-bcdb-45d3-b51d-f3e84f999973"). InnerVolumeSpecName "kube-api-access-gbkzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:50:41.869467 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.869385    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f517b39f-bcdb-45d3-b51d-f3e84f999973-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f517b39f-bcdb-45d3-b51d-f3e84f999973" (UID: "f517b39f-bcdb-45d3-b51d-f3e84f999973"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:50:41.869467 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.869395    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-dshm" (OuterVolumeSpecName: "dshm") pod "f517b39f-bcdb-45d3-b51d-f3e84f999973" (UID: "f517b39f-bcdb-45d3-b51d-f3e84f999973"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:50:41.968076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.968035    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f517b39f-bcdb-45d3-b51d-f3e84f999973-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:41.968076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.968073    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbkzk\" (UniqueName: \"kubernetes.io/projected/f517b39f-bcdb-45d3-b51d-f3e84f999973-kube-api-access-gbkzk\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:41.968076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.968085    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:41.968674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.968093    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:41.968674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:41.968102    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f517b39f-bcdb-45d3-b51d-f3e84f999973-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:50:42.300780 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:42.300695    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"
Sep 05 02:50:42.300780 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:42.300726    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx" event={"ID":"f517b39f-bcdb-45d3-b51d-f3e84f999973","Type":"ContainerDied","Data":"3261614ae55b212815abc68fce248fb08d4e2cdb0804eabbe359bfdba9eb34f2"}
Sep 05 02:50:42.300970 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:42.300789    2557 scope.go:117] "RemoveContainer" containerID="8921015dc52c99467133bb76296f4a42f5e2fd4de2807885e9f0f6abad37a89b"
Sep 05 02:50:42.322610 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:42.322567    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"]
Sep 05 02:50:42.326348 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:42.326320    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445kmkx"]
Sep 05 02:50:42.990861 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:42.990827    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f517b39f-bcdb-45d3-b51d-f3e84f999973" path="/var/lib/kubelet/pods/f517b39f-bcdb-45d3-b51d-f3e84f999973/volumes"
Sep 05 02:50:53.615814 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:53.615778    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"]
Sep 05 02:50:53.685782 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:50:53.685729    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:08.635194 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:08.635151    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj" podUID="b825918b-5c03-4dc5-afe8-e5dc755509f7" containerName="main" containerID="cri-o://49148f36676bde811a15629dd9bbabb53b512619977e446dcdb5b9de5ad052c5" gracePeriod=45
Sep 05 02:51:08.884744 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:08.884721    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:51:08.904522 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:08.904424    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z55fp\" (UniqueName: \"kubernetes.io/projected/b825918b-5c03-4dc5-afe8-e5dc755509f7-kube-api-access-z55fp\") pod \"b825918b-5c03-4dc5-afe8-e5dc755509f7\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") "
Sep 05 02:51:08.904522 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:08.904503    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-dshm\") pod \"b825918b-5c03-4dc5-afe8-e5dc755509f7\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") "
Sep 05 02:51:08.904723 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:08.904568    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b825918b-5c03-4dc5-afe8-e5dc755509f7-tls-certs\") pod \"b825918b-5c03-4dc5-afe8-e5dc755509f7\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") "
Sep 05 02:51:08.904723 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:08.904636    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-home\") pod \"b825918b-5c03-4dc5-afe8-e5dc755509f7\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") "
Sep 05 02:51:08.904723 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:08.904680    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-tmp-dir\") pod \"b825918b-5c03-4dc5-afe8-e5dc755509f7\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") "
Sep 05 02:51:08.904723 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:08.904703    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-model-cache\") pod \"b825918b-5c03-4dc5-afe8-e5dc755509f7\" (UID: \"b825918b-5c03-4dc5-afe8-e5dc755509f7\") "
Sep 05 02:51:08.905373 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:08.905127    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b825918b-5c03-4dc5-afe8-e5dc755509f7" (UID: "b825918b-5c03-4dc5-afe8-e5dc755509f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:51:08.905373 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:08.905207    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "b825918b-5c03-4dc5-afe8-e5dc755509f7" (UID: "b825918b-5c03-4dc5-afe8-e5dc755509f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:51:08.905373 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:08.905190    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-home" (OuterVolumeSpecName: "home") pod "b825918b-5c03-4dc5-afe8-e5dc755509f7" (UID: "b825918b-5c03-4dc5-afe8-e5dc755509f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:51:08.907168 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:08.907145    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b825918b-5c03-4dc5-afe8-e5dc755509f7-kube-api-access-z55fp" (OuterVolumeSpecName: "kube-api-access-z55fp") pod "b825918b-5c03-4dc5-afe8-e5dc755509f7" (UID: "b825918b-5c03-4dc5-afe8-e5dc755509f7"). InnerVolumeSpecName "kube-api-access-z55fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:51:08.907318 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:08.907300    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-dshm" (OuterVolumeSpecName: "dshm") pod "b825918b-5c03-4dc5-afe8-e5dc755509f7" (UID: "b825918b-5c03-4dc5-afe8-e5dc755509f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:51:08.907583 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:08.907556    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b825918b-5c03-4dc5-afe8-e5dc755509f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b825918b-5c03-4dc5-afe8-e5dc755509f7" (UID: "b825918b-5c03-4dc5-afe8-e5dc755509f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:51:09.006241 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.006215    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:51:09.006241 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.006240    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b825918b-5c03-4dc5-afe8-e5dc755509f7-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:51:09.006459 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.006250    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:51:09.006459 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.006258    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:51:09.006459 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.006266    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b825918b-5c03-4dc5-afe8-e5dc755509f7-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:51:09.006459 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.006276    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z55fp\" (UniqueName: \"kubernetes.io/projected/b825918b-5c03-4dc5-afe8-e5dc755509f7-kube-api-access-z55fp\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:51:09.450045 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.450003    2557 generic.go:356] "Generic (PLEG): container finished" podID="b825918b-5c03-4dc5-afe8-e5dc755509f7" containerID="49148f36676bde811a15629dd9bbabb53b512619977e446dcdb5b9de5ad052c5" exitCode=0
Sep 05 02:51:09.450242 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.450063    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"
Sep 05 02:51:09.450242 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.450088    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj" event={"ID":"b825918b-5c03-4dc5-afe8-e5dc755509f7","Type":"ContainerDied","Data":"49148f36676bde811a15629dd9bbabb53b512619977e446dcdb5b9de5ad052c5"}
Sep 05 02:51:09.450242 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.450164    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj" event={"ID":"b825918b-5c03-4dc5-afe8-e5dc755509f7","Type":"ContainerDied","Data":"a2585ec885a387629731296a44bdf1b19d9fcb6c7c65a44ad9d323fb4a43540e"}
Sep 05 02:51:09.450242 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.450198    2557 scope.go:117] "RemoveContainer" containerID="49148f36676bde811a15629dd9bbabb53b512619977e446dcdb5b9de5ad052c5"
Sep 05 02:51:09.459154 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.459132    2557 scope.go:117] "RemoveContainer" containerID="49148f36676bde811a15629dd9bbabb53b512619977e446dcdb5b9de5ad052c5"
Sep 05 02:51:09.459412 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:51:09.459394    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49148f36676bde811a15629dd9bbabb53b512619977e446dcdb5b9de5ad052c5\": container with ID starting with 49148f36676bde811a15629dd9bbabb53b512619977e446dcdb5b9de5ad052c5 not found: ID does not exist" containerID="49148f36676bde811a15629dd9bbabb53b512619977e446dcdb5b9de5ad052c5"
Sep 05 02:51:09.459476 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.459418    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49148f36676bde811a15629dd9bbabb53b512619977e446dcdb5b9de5ad052c5"} err="failed to get container status \"49148f36676bde811a15629dd9bbabb53b512619977e446dcdb5b9de5ad052c5\": rpc error: code = NotFound desc = could not find container \"49148f36676bde811a15629dd9bbabb53b512619977e446dcdb5b9de5ad052c5\": container with ID starting with 49148f36676bde811a15629dd9bbabb53b512619977e446dcdb5b9de5ad052c5 not found: ID does not exist"
Sep 05 02:51:09.465933 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.465908    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"]
Sep 05 02:51:09.470022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:09.470001    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lx5sj"]
Sep 05 02:51:10.990890 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:10.990856    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b825918b-5c03-4dc5-afe8-e5dc755509f7" path="/var/lib/kubelet/pods/b825918b-5c03-4dc5-afe8-e5dc755509f7/volumes"
Sep 05 02:51:15.384491 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:15.384432    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:15.485619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:15.485577    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:15.600725 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:15.600696    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:20.368897 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:20.368857    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"]
Sep 05 02:51:20.461717 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:20.461685    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:26.419018 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:26.418983    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:29.345652 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:29.345602    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:29.554385 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:29.554344    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:29.686885 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:29.686799    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:35.388131 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.388064    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v" podUID="56c84102-ed66-4be6-b289-639ad5c62c83" containerName="main" containerID="cri-o://701e6105feef2d3d2a113d597337d6979edfe0f9406e7a35888001c68d1b6233" gracePeriod=45
Sep 05 02:51:35.596562 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.596527    2557 generic.go:356] "Generic (PLEG): container finished" podID="56c84102-ed66-4be6-b289-639ad5c62c83" containerID="701e6105feef2d3d2a113d597337d6979edfe0f9406e7a35888001c68d1b6233" exitCode=0
Sep 05 02:51:35.596718 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.596601    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v" event={"ID":"56c84102-ed66-4be6-b289-639ad5c62c83","Type":"ContainerDied","Data":"701e6105feef2d3d2a113d597337d6979edfe0f9406e7a35888001c68d1b6233"}
Sep 05 02:51:35.630706 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.630629    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:35.638343 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.632505    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:51:35.744231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.744196    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56c84102-ed66-4be6-b289-639ad5c62c83-tls-certs\") pod \"56c84102-ed66-4be6-b289-639ad5c62c83\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") "
Sep 05 02:51:35.744423 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.744246    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-tmp-dir\") pod \"56c84102-ed66-4be6-b289-639ad5c62c83\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") "
Sep 05 02:51:35.744423 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.744265    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-dshm\") pod \"56c84102-ed66-4be6-b289-639ad5c62c83\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") "
Sep 05 02:51:35.744423 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.744292    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-model-cache\") pod \"56c84102-ed66-4be6-b289-639ad5c62c83\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") "
Sep 05 02:51:35.744423 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.744333    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-home\") pod \"56c84102-ed66-4be6-b289-639ad5c62c83\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") "
Sep 05 02:51:35.744423 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.744377    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq6q4\" (UniqueName: \"kubernetes.io/projected/56c84102-ed66-4be6-b289-639ad5c62c83-kube-api-access-wq6q4\") pod \"56c84102-ed66-4be6-b289-639ad5c62c83\" (UID: \"56c84102-ed66-4be6-b289-639ad5c62c83\") "
Sep 05 02:51:35.744753 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.744578    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56c84102-ed66-4be6-b289-639ad5c62c83" (UID: "56c84102-ed66-4be6-b289-639ad5c62c83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:51:35.744753 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.744598    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-model-cache" (OuterVolumeSpecName: "model-cache") pod "56c84102-ed66-4be6-b289-639ad5c62c83" (UID: "56c84102-ed66-4be6-b289-639ad5c62c83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:51:35.744753 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.744687    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-home" (OuterVolumeSpecName: "home") pod "56c84102-ed66-4be6-b289-639ad5c62c83" (UID: "56c84102-ed66-4be6-b289-639ad5c62c83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:51:35.746426 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.746394    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56c84102-ed66-4be6-b289-639ad5c62c83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56c84102-ed66-4be6-b289-639ad5c62c83" (UID: "56c84102-ed66-4be6-b289-639ad5c62c83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:51:35.746546 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.746449    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c84102-ed66-4be6-b289-639ad5c62c83-kube-api-access-wq6q4" (OuterVolumeSpecName: "kube-api-access-wq6q4") pod "56c84102-ed66-4be6-b289-639ad5c62c83" (UID: "56c84102-ed66-4be6-b289-639ad5c62c83"). InnerVolumeSpecName "kube-api-access-wq6q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:51:35.746546 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.746468    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-dshm" (OuterVolumeSpecName: "dshm") pod "56c84102-ed66-4be6-b289-639ad5c62c83" (UID: "56c84102-ed66-4be6-b289-639ad5c62c83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:51:35.845286 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.845252    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq6q4\" (UniqueName: \"kubernetes.io/projected/56c84102-ed66-4be6-b289-639ad5c62c83-kube-api-access-wq6q4\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:51:35.845286 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.845286    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56c84102-ed66-4be6-b289-639ad5c62c83-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:51:35.845480 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.845323    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:51:35.845480 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.845336    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:51:35.845480 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.845349    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:51:35.845480 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:35.845361    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56c84102-ed66-4be6-b289-639ad5c62c83-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:51:36.603841 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.603805    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v" event={"ID":"56c84102-ed66-4be6-b289-639ad5c62c83","Type":"ContainerDied","Data":"245f02df2497e674eb37bb05dcad4d762290467c3ff1025e0950faf1f7bb8f22"}
Sep 05 02:51:36.604318 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.603866    2557 scope.go:117] "RemoveContainer" containerID="701e6105feef2d3d2a113d597337d6979edfe0f9406e7a35888001c68d1b6233"
Sep 05 02:51:36.604318 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.603822    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"
Sep 05 02:51:36.651118 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.651079    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"]
Sep 05 02:51:36.654261 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.654236    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-9vz5v"]
Sep 05 02:51:36.819048 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.819014    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"]
Sep 05 02:51:36.819401 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.819386    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b825918b-5c03-4dc5-afe8-e5dc755509f7" containerName="main"
Sep 05 02:51:36.819401 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.819401    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b825918b-5c03-4dc5-afe8-e5dc755509f7" containerName="main"
Sep 05 02:51:36.819559 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.819409    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c84102-ed66-4be6-b289-639ad5c62c83" containerName="main"
Sep 05 02:51:36.819559 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.819414    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c84102-ed66-4be6-b289-639ad5c62c83" containerName="main"
Sep 05 02:51:36.819559 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.819431    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f517b39f-bcdb-45d3-b51d-f3e84f999973" containerName="main"
Sep 05 02:51:36.819559 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.819455    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f517b39f-bcdb-45d3-b51d-f3e84f999973" containerName="main"
Sep 05 02:51:36.819559 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.819520    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="56c84102-ed66-4be6-b289-639ad5c62c83" containerName="main"
Sep 05 02:51:36.819559 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.819527    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f517b39f-bcdb-45d3-b51d-f3e84f999973" containerName="main"
Sep 05 02:51:36.819559 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.819534    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b825918b-5c03-4dc5-afe8-e5dc755509f7" containerName="main"
Sep 05 02:51:36.823881 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.823859    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:36.827717 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.827689    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 05 02:51:36.828141 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.828117    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 05 02:51:36.828361 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.828155    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-qrchx\""
Sep 05 02:51:36.828543 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.828527    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 05 02:51:36.838776 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.838748    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"]
Sep 05 02:51:36.847362 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.847336    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:36.955127 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.955030    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-tmp-dir\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:36.955127 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.955083    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a948aff3-a857-4abc-8197-9e48b01043eb-tls-certs\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:36.955324 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.955136    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8rlp\" (UniqueName: \"kubernetes.io/projected/a948aff3-a857-4abc-8197-9e48b01043eb-kube-api-access-l8rlp\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:36.955324 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.955166    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-model-cache\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:36.955324 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.955198    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-home\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:36.955324 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.955257    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-dshm\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:36.991035 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:36.991004    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56c84102-ed66-4be6-b289-639ad5c62c83" path="/var/lib/kubelet/pods/56c84102-ed66-4be6-b289-639ad5c62c83/volumes"
Sep 05 02:51:37.056139 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.056100    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-tmp-dir\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:37.056292 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.056146    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a948aff3-a857-4abc-8197-9e48b01043eb-tls-certs\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:37.056292 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.056175    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8rlp\" (UniqueName: \"kubernetes.io/projected/a948aff3-a857-4abc-8197-9e48b01043eb-kube-api-access-l8rlp\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:37.056292 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.056203    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-model-cache\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:37.056292 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.056225    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-home\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:37.056292 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.056267    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-dshm\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:37.056601 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.056574    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-tmp-dir\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:37.056683 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.056629    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-model-cache\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:37.056683 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.056665    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-home\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:37.058560 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.058537    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-dshm\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:37.058674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.058642    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a948aff3-a857-4abc-8197-9e48b01043eb-tls-certs\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:37.064071 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.064044    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8rlp\" (UniqueName: \"kubernetes.io/projected/a948aff3-a857-4abc-8197-9e48b01043eb-kube-api-access-l8rlp\") pod \"decom-v2-kserve-77f867d787-9gvcf\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:37.134233 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.134199    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:37.421025 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.420990    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"]
Sep 05 02:51:37.423579 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:51:37.423546    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda948aff3_a857_4abc_8197_9e48b01043eb.slice/crio-1e3ba5d08e5eb91bf7fbc286b146efa27c6ff291fb1063918cf3a4eecd2bbb76 WatchSource:0}: Error finding container 1e3ba5d08e5eb91bf7fbc286b146efa27c6ff291fb1063918cf3a4eecd2bbb76: Status 404 returned error can't find the container with id 1e3ba5d08e5eb91bf7fbc286b146efa27c6ff291fb1063918cf3a4eecd2bbb76
Sep 05 02:51:37.612310 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.612277    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf" event={"ID":"a948aff3-a857-4abc-8197-9e48b01043eb","Type":"ContainerStarted","Data":"c8a3cc471860f49def84f85001d413520e72ac5d6f6403bc3bf5ec6032bbc0d1"}
Sep 05 02:51:37.612741 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.612329    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf" event={"ID":"a948aff3-a857-4abc-8197-9e48b01043eb","Type":"ContainerStarted","Data":"1e3ba5d08e5eb91bf7fbc286b146efa27c6ff291fb1063918cf3a4eecd2bbb76"}
Sep 05 02:51:37.631684 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:37.631622    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf" podStartSLOduration=1.631607449 podStartE2EDuration="1.631607449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:51:37.631034133 +0000 UTC m=+2001.186190453" watchObservedRunningTime="2026-09-05 02:51:37.631607449 +0000 UTC m=+2001.186763758"
Sep 05 02:51:38.134694 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:38.134653    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:41.569863 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:41.569824    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:41.670404 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:41.670364    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:47.134795 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:47.134759    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:47.148991 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:47.148959    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:47.685821 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:47.685785    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:51:54.188388 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.188353    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"]
Sep 05 02:51:54.198446 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.198400    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.200808 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.200777    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 05 02:51:54.200967 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.200809    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:51:54.201681 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.201645    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-tfjbj\""
Sep 05 02:51:54.201681 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.201666    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 05 02:51:54.201848 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.201686    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 05 02:51:54.201848 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.201669    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 05 02:51:54.205675 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.205651    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"]
Sep 05 02:51:54.205792 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.205660    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-model-cache\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.205792 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.205741    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.205912 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.205792    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-home\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.205912 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.205817    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.205912 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.205842    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.206029 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.205934    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfdz5\" (UniqueName: \"kubernetes.io/projected/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-kube-api-access-xfdz5\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.206085 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.206036    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.206135 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.206098    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-dshm\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.253646 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.253603    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:54.307075 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.307017    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfdz5\" (UniqueName: \"kubernetes.io/projected/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-kube-api-access-xfdz5\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.307264 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.307128    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.307264 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.307161    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-dshm\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.307264 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.307184    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-model-cache\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.307264 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.307203    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.307264 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.307255    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-home\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.307567 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.307279    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.307567 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.307304    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.307782 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.307757    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.307944 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.307911    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.307944 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.307795    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-model-cache\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.308134 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.307838    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-home\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.308134 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.308067    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.309684 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.309661    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-dshm\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.309932 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.309916    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.314652 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.314625    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfdz5\" (UniqueName: \"kubernetes.io/projected/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-kube-api-access-xfdz5\") pod \"tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.457395 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.457314    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"]
Sep 05 02:51:54.462129 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.462105    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:51:54.474565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.474493    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"]
Sep 05 02:51:54.497875 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.497838    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:54.510049 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.510019    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d11ff62-62d9-4300-85aa-d59d3741d587-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:51:54.510227 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.510057    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d11ff62-62d9-4300-85aa-d59d3741d587-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:51:54.510227 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.510086    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d11ff62-62d9-4300-85aa-d59d3741d587-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:51:54.510351 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.510253    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d11ff62-62d9-4300-85aa-d59d3741d587-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:51:54.510351 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.510287    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:51:54.554465 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.554418    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:54.611575 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.611545    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d11ff62-62d9-4300-85aa-d59d3741d587-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:51:54.611730 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.611633    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d11ff62-62d9-4300-85aa-d59d3741d587-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:51:54.611730 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.611674    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d11ff62-62d9-4300-85aa-d59d3741d587-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:51:54.611730 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.611695    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d11ff62-62d9-4300-85aa-d59d3741d587-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:51:54.612097 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.612075    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d11ff62-62d9-4300-85aa-d59d3741d587-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:51:54.612179 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.612126    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d11ff62-62d9-4300-85aa-d59d3741d587-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:51:54.612456 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.612419    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d11ff62-62d9-4300-85aa-d59d3741d587-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:51:54.614184 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.614162    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d11ff62-62d9-4300-85aa-d59d3741d587-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:51:54.775893 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.775814    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:51:54.788869 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:54.788760    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"]
Sep 05 02:51:54.790531 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:51:54.790505    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85de0cf_2cd7_45ae_8dfa_317fd7bfc07b.slice/crio-a4ff207142abce4d9789f26fe7ce17ed0581709e9c064f9f8771fedf42656a6c WatchSource:0}: Error finding container a4ff207142abce4d9789f26fe7ce17ed0581709e9c064f9f8771fedf42656a6c: Status 404 returned error can't find the container with id a4ff207142abce4d9789f26fe7ce17ed0581709e9c064f9f8771fedf42656a6c
Sep 05 02:51:55.059285 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:55.059253    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"]
Sep 05 02:51:55.059786 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:51:55.059762    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d11ff62_62d9_4300_85aa_d59d3741d587.slice/crio-96689246a64c6c4b3dd96b4d1dc2b54398d2b266b5c970f2ef4477db948e4ff7 WatchSource:0}: Error finding container 96689246a64c6c4b3dd96b4d1dc2b54398d2b266b5c970f2ef4477db948e4ff7: Status 404 returned error can't find the container with id 96689246a64c6c4b3dd96b4d1dc2b54398d2b266b5c970f2ef4477db948e4ff7
Sep 05 02:51:55.716999 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:55.716952    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv" event={"ID":"7d11ff62-62d9-4300-85aa-d59d3741d587","Type":"ContainerStarted","Data":"de5a5fe1ac008bebe213b934164e9f84ed0209bbbc132aa1cf34c9fb57185d12"}
Sep 05 02:51:55.717486 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:55.717029    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv" event={"ID":"7d11ff62-62d9-4300-85aa-d59d3741d587","Type":"ContainerStarted","Data":"96689246a64c6c4b3dd96b4d1dc2b54398d2b266b5c970f2ef4477db948e4ff7"}
Sep 05 02:51:55.718786 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:55.718760    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j" event={"ID":"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b","Type":"ContainerStarted","Data":"7c5b06a3b19dc8bae371fb5d07ed8ede6a84f872abd11083171788f2cca80f40"}
Sep 05 02:51:55.718905 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:55.718810    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j" event={"ID":"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b","Type":"ContainerStarted","Data":"a4ff207142abce4d9789f26fe7ce17ed0581709e9c064f9f8771fedf42656a6c"}
Sep 05 02:51:57.581617 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:57.581572    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:57.732225 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:57.732140    2557 generic.go:356] "Generic (PLEG): container finished" podID="c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" containerID="7c5b06a3b19dc8bae371fb5d07ed8ede6a84f872abd11083171788f2cca80f40" exitCode=0
Sep 05 02:51:57.732225 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:57.732214    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j" event={"ID":"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b","Type":"ContainerDied","Data":"7c5b06a3b19dc8bae371fb5d07ed8ede6a84f872abd11083171788f2cca80f40"}
Sep 05 02:51:58.738953 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:58.738911    2557 generic.go:356] "Generic (PLEG): container finished" podID="7d11ff62-62d9-4300-85aa-d59d3741d587" containerID="de5a5fe1ac008bebe213b934164e9f84ed0209bbbc132aa1cf34c9fb57185d12" exitCode=0
Sep 05 02:51:58.739485 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:58.739010    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv" event={"ID":"7d11ff62-62d9-4300-85aa-d59d3741d587","Type":"ContainerDied","Data":"de5a5fe1ac008bebe213b934164e9f84ed0209bbbc132aa1cf34c9fb57185d12"}
Sep 05 02:51:58.740854 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:58.740827    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:51:58.744358 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:58.744332    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j" event={"ID":"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b","Type":"ContainerStarted","Data":"3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45"}
Sep 05 02:51:58.775688 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:58.775642    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j" podStartSLOduration=2.159798489 podStartE2EDuration="4.775628209s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.61582972s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:51:58.774667779 +0000 UTC m=+2022.329824090" watchObservedRunningTime="2026-09-05 02:51:58.775628209 +0000 UTC m=+2022.330784536"
Sep 05 02:51:59.290575 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:59.290527    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:51:59.510629 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:51:59.510597    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:52:04.511585 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:52:04.511476    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:52:04.531957 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:52:04.531925    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:52:04.815756 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:52:04.815427    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:52:17.727396 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:52:17.727328    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:52:17.829734 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:52:17.829674    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:52:49.109619 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:52:49.109583    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv" event={"ID":"7d11ff62-62d9-4300-85aa-d59d3741d587","Type":"ContainerStarted","Data":"660e4f9bb696c25978e3ca68017b7ebb33ef5ed7db3cf2e05b324fc1e9b7de88"}
Sep 05 02:52:49.110081 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:52:49.109801    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:52:49.130075 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:52:49.130026    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv" podStartSLOduration=3.202296874 podStartE2EDuration="55.130011578s" totalImagesPullingTime="49.323327552s" totalInitContainerRuntime="2.604387152s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:52:49.127145776 +0000 UTC m=+2072.682302087" watchObservedRunningTime="2026-09-05 02:52:49.130011578 +0000 UTC m=+2072.685167923"
Sep 05 02:52:52.365021 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:52:52.364984    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:52:56.299605 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:52:56.299560    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"]
Sep 05 02:52:56.322129 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:52:56.322095    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:53:11.318984 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:11.318943    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf" podUID="a948aff3-a857-4abc-8197-9e48b01043eb" containerName="main" containerID="cri-o://c8a3cc471860f49def84f85001d413520e72ac5d6f6403bc3bf5ec6032bbc0d1" gracePeriod=45
Sep 05 02:53:11.671430 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:11.671328    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf" podUID="a948aff3-a857-4abc-8197-9e48b01043eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 05 02:53:12.246144 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.246053    2557 generic.go:356] "Generic (PLEG): container finished" podID="a948aff3-a857-4abc-8197-9e48b01043eb" containerID="c8a3cc471860f49def84f85001d413520e72ac5d6f6403bc3bf5ec6032bbc0d1" exitCode=0
Sep 05 02:53:12.246294 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.246130    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf" event={"ID":"a948aff3-a857-4abc-8197-9e48b01043eb","Type":"ContainerDied","Data":"c8a3cc471860f49def84f85001d413520e72ac5d6f6403bc3bf5ec6032bbc0d1"}
Sep 05 02:53:12.691951 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.691914    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:53:12.764386 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.764339    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-home\") pod \"a948aff3-a857-4abc-8197-9e48b01043eb\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") "
Sep 05 02:53:12.764568 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.764463    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a948aff3-a857-4abc-8197-9e48b01043eb-tls-certs\") pod \"a948aff3-a857-4abc-8197-9e48b01043eb\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") "
Sep 05 02:53:12.764568 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.764498    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-model-cache\") pod \"a948aff3-a857-4abc-8197-9e48b01043eb\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") "
Sep 05 02:53:12.764568 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.764551    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8rlp\" (UniqueName: \"kubernetes.io/projected/a948aff3-a857-4abc-8197-9e48b01043eb-kube-api-access-l8rlp\") pod \"a948aff3-a857-4abc-8197-9e48b01043eb\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") "
Sep 05 02:53:12.764699 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.764580    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-dshm\") pod \"a948aff3-a857-4abc-8197-9e48b01043eb\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") "
Sep 05 02:53:12.764699 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.764617    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-tmp-dir\") pod \"a948aff3-a857-4abc-8197-9e48b01043eb\" (UID: \"a948aff3-a857-4abc-8197-9e48b01043eb\") "
Sep 05 02:53:12.765154 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.765126    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a948aff3-a857-4abc-8197-9e48b01043eb" (UID: "a948aff3-a857-4abc-8197-9e48b01043eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:53:12.765630 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.765604    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-home" (OuterVolumeSpecName: "home") pod "a948aff3-a857-4abc-8197-9e48b01043eb" (UID: "a948aff3-a857-4abc-8197-9e48b01043eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:53:12.768093 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.768068    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "a948aff3-a857-4abc-8197-9e48b01043eb" (UID: "a948aff3-a857-4abc-8197-9e48b01043eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:53:12.770020 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.769994    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a948aff3-a857-4abc-8197-9e48b01043eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a948aff3-a857-4abc-8197-9e48b01043eb" (UID: "a948aff3-a857-4abc-8197-9e48b01043eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:53:12.771381 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.771348    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a948aff3-a857-4abc-8197-9e48b01043eb-kube-api-access-l8rlp" (OuterVolumeSpecName: "kube-api-access-l8rlp") pod "a948aff3-a857-4abc-8197-9e48b01043eb" (UID: "a948aff3-a857-4abc-8197-9e48b01043eb"). InnerVolumeSpecName "kube-api-access-l8rlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:53:12.771764 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.771708    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-dshm" (OuterVolumeSpecName: "dshm") pod "a948aff3-a857-4abc-8197-9e48b01043eb" (UID: "a948aff3-a857-4abc-8197-9e48b01043eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:53:12.851793 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.851760    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:53:12.865940 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.865915    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:53:12.865940 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.865940    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a948aff3-a857-4abc-8197-9e48b01043eb-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:53:12.866127 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.865951    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:53:12.866127 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.865961    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8rlp\" (UniqueName: \"kubernetes.io/projected/a948aff3-a857-4abc-8197-9e48b01043eb-kube-api-access-l8rlp\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:53:12.866127 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.865973    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:53:12.866127 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.865988    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a948aff3-a857-4abc-8197-9e48b01043eb-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:53:12.954188 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:12.954142    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:53:13.254227 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:13.254150    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"
Sep 05 02:53:13.254227 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:13.254174    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf" event={"ID":"a948aff3-a857-4abc-8197-9e48b01043eb","Type":"ContainerDied","Data":"1e3ba5d08e5eb91bf7fbc286b146efa27c6ff291fb1063918cf3a4eecd2bbb76"}
Sep 05 02:53:13.254433 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:13.254247    2557 scope.go:117] "RemoveContainer" containerID="c8a3cc471860f49def84f85001d413520e72ac5d6f6403bc3bf5ec6032bbc0d1"
Sep 05 02:53:13.272394 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:13.272368    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"]
Sep 05 02:53:13.277465 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:13.277421    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-9gvcf"]
Sep 05 02:53:14.990736 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:14.990701    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a948aff3-a857-4abc-8197-9e48b01043eb" path="/var/lib/kubelet/pods/a948aff3-a857-4abc-8197-9e48b01043eb/volumes"
Sep 05 02:53:17.085228 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:17.085200    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:53:17.092824 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:17.092800    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:53:17.839610 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:17.839578    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:53:20.126498 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:20.126461    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:53:33.584076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:33.584030    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:53:41.436888 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:41.436850    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:53:44.129525 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:44.129457    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv_7d11ff62-62d9-4300-85aa-d59d3741d587/main/0.log"
Sep 05 02:53:55.141984 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:53:55.141940    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:00.623669 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:00.623630    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:00.724303 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:00.724266    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:05.028457 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.028396    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:05.056070 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.056031    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"]
Sep 05 02:54:05.056392 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.056354    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv" podUID="7d11ff62-62d9-4300-85aa-d59d3741d587" containerName="main" containerID="cri-o://660e4f9bb696c25978e3ca68017b7ebb33ef5ed7db3cf2e05b324fc1e9b7de88" gracePeriod=30
Sep 05 02:54:05.069476 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.069427    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"]
Sep 05 02:54:05.078054 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:54:05.078020    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:54:05.078054 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:54:05.078054 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:54:05.078054 ip-10-0-143-128 kubenswrapper[2557]:  > containerID="3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45" cmd=["/bin/sleep","15"]
Sep 05 02:54:05.078297 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.078074    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j" message="" err=<
Sep 05 02:54:05.078297 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:54:05.078297 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:54:05.078297 ip-10-0-143-128 kubenswrapper[2557]:  >
Sep 05 02:54:05.078297 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:54:05.078087    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:54:05.078297 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:54:05.078297 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:54:05.078297 ip-10-0-143-128 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j" podUID="c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" containerName="main" containerID="cri-o://3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45"
Sep 05 02:54:05.078297 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.078128    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j" podUID="c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" containerName="main" containerID="cri-o://3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45" gracePeriod=60
Sep 05 02:54:05.144282 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.144245    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:05.447124 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.447095    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:54:05.542076 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.542043    2557 generic.go:356] "Generic (PLEG): container finished" podID="c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" containerID="3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45" exitCode=0
Sep 05 02:54:05.542215 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.542090    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"
Sep 05 02:54:05.542215 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.542126    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j" event={"ID":"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b","Type":"ContainerDied","Data":"3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45"}
Sep 05 02:54:05.542215 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.542181    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j" event={"ID":"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b","Type":"ContainerDied","Data":"a4ff207142abce4d9789f26fe7ce17ed0581709e9c064f9f8771fedf42656a6c"}
Sep 05 02:54:05.542215 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.542208    2557 scope.go:117] "RemoveContainer" containerID="3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45"
Sep 05 02:54:05.543222 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.543200    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-home\") pod \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") "
Sep 05 02:54:05.543357 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.543239    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-tls-certs\") pod \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") "
Sep 05 02:54:05.543357 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.543265    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfdz5\" (UniqueName: \"kubernetes.io/projected/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-kube-api-access-xfdz5\") pod \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") "
Sep 05 02:54:05.543577 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.543531    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-home" (OuterVolumeSpecName: "home") pod "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" (UID: "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:05.543716 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.543690    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-tmp-dir\") pod \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") "
Sep 05 02:54:05.543842 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.543778    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-model-cache\") pod \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") "
Sep 05 02:54:05.543842 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.543804    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-cabundle-cert\") pod \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") "
Sep 05 02:54:05.543842 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.543837    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-dshm\") pod \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") "
Sep 05 02:54:05.544063 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.543867    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-kserve-provision-location\") pod \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\" (UID: \"c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b\") "
Sep 05 02:54:05.544183 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.544161    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.544842 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.544537    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" (UID: "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:54:05.544842 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.544752    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-model-cache" (OuterVolumeSpecName: "model-cache") pod "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" (UID: "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:05.544842 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.544809    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" (UID: "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:05.545958 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.545886    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-kube-api-access-xfdz5" (OuterVolumeSpecName: "kube-api-access-xfdz5") pod "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" (UID: "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b"). InnerVolumeSpecName "kube-api-access-xfdz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:54:05.546068 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.546012    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" (UID: "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:54:05.547044 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.547018    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-dshm" (OuterVolumeSpecName: "dshm") pod "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" (UID: "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:05.566048 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.566018    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" (UID: "c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:05.568750 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.568729    2557 scope.go:117] "RemoveContainer" containerID="7c5b06a3b19dc8bae371fb5d07ed8ede6a84f872abd11083171788f2cca80f40"
Sep 05 02:54:05.576682 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.576654    2557 scope.go:117] "RemoveContainer" containerID="3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45"
Sep 05 02:54:05.576959 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:54:05.576935    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45\": container with ID starting with 3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45 not found: ID does not exist" containerID="3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45"
Sep 05 02:54:05.577012 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.576968    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45"} err="failed to get container status \"3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45\": rpc error: code = NotFound desc = could not find container \"3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45\": container with ID starting with 3383200af125c4738664c8c18cf0fd73766bee10b71179740562122b88242e45 not found: ID does not exist"
Sep 05 02:54:05.577012 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.576985    2557 scope.go:117] "RemoveContainer" containerID="7c5b06a3b19dc8bae371fb5d07ed8ede6a84f872abd11083171788f2cca80f40"
Sep 05 02:54:05.577250 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:54:05.577232    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5b06a3b19dc8bae371fb5d07ed8ede6a84f872abd11083171788f2cca80f40\": container with ID starting with 7c5b06a3b19dc8bae371fb5d07ed8ede6a84f872abd11083171788f2cca80f40 not found: ID does not exist" containerID="7c5b06a3b19dc8bae371fb5d07ed8ede6a84f872abd11083171788f2cca80f40"
Sep 05 02:54:05.577329 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.577250    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5b06a3b19dc8bae371fb5d07ed8ede6a84f872abd11083171788f2cca80f40"} err="failed to get container status \"7c5b06a3b19dc8bae371fb5d07ed8ede6a84f872abd11083171788f2cca80f40\": rpc error: code = NotFound desc = could not find container \"7c5b06a3b19dc8bae371fb5d07ed8ede6a84f872abd11083171788f2cca80f40\": container with ID starting with 7c5b06a3b19dc8bae371fb5d07ed8ede6a84f872abd11083171788f2cca80f40 not found: ID does not exist"
Sep 05 02:54:05.644759 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.644724    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.644759 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.644758    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.644988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.644771    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.644988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.644783    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.644988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.644796    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.644988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.644809    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.644988 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.644823    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfdz5\" (UniqueName: \"kubernetes.io/projected/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b-kube-api-access-xfdz5\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.691193 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.691168    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:54:05.846200 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.846166    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d11ff62-62d9-4300-85aa-d59d3741d587-tmp-dir\") pod \"7d11ff62-62d9-4300-85aa-d59d3741d587\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") "
Sep 05 02:54:05.846354 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.846303    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d11ff62-62d9-4300-85aa-d59d3741d587-cabundle-cert\") pod \"7d11ff62-62d9-4300-85aa-d59d3741d587\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") "
Sep 05 02:54:05.846354 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.846349    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d11ff62-62d9-4300-85aa-d59d3741d587-kserve-provision-location\") pod \"7d11ff62-62d9-4300-85aa-d59d3741d587\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") "
Sep 05 02:54:05.846517 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.846380    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d11ff62-62d9-4300-85aa-d59d3741d587-tls-certs\") pod \"7d11ff62-62d9-4300-85aa-d59d3741d587\" (UID: \"7d11ff62-62d9-4300-85aa-d59d3741d587\") "
Sep 05 02:54:05.846703 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.846676    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d11ff62-62d9-4300-85aa-d59d3741d587-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d11ff62-62d9-4300-85aa-d59d3741d587" (UID: "7d11ff62-62d9-4300-85aa-d59d3741d587"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:05.846767 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.846718    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d11ff62-62d9-4300-85aa-d59d3741d587-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d11ff62-62d9-4300-85aa-d59d3741d587" (UID: "7d11ff62-62d9-4300-85aa-d59d3741d587"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:54:05.848607 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.848576    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d11ff62-62d9-4300-85aa-d59d3741d587-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d11ff62-62d9-4300-85aa-d59d3741d587" (UID: "7d11ff62-62d9-4300-85aa-d59d3741d587"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:54:05.867795 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.867771    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d11ff62-62d9-4300-85aa-d59d3741d587-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d11ff62-62d9-4300-85aa-d59d3741d587" (UID: "7d11ff62-62d9-4300-85aa-d59d3741d587"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:54:05.882172 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.882147    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"]
Sep 05 02:54:05.891870 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.891849    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b99f8875c-g5w4j"]
Sep 05 02:54:05.947621 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.947592    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d11ff62-62d9-4300-85aa-d59d3741d587-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.947621 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.947615    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d11ff62-62d9-4300-85aa-d59d3741d587-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.947621 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.947626    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d11ff62-62d9-4300-85aa-d59d3741d587-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:54:05.947807 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:05.947635    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d11ff62-62d9-4300-85aa-d59d3741d587-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:54:06.550361 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:06.550278    2557 generic.go:356] "Generic (PLEG): container finished" podID="7d11ff62-62d9-4300-85aa-d59d3741d587" containerID="660e4f9bb696c25978e3ca68017b7ebb33ef5ed7db3cf2e05b324fc1e9b7de88" exitCode=0
Sep 05 02:54:06.550361 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:06.550330    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"
Sep 05 02:54:06.550858 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:06.550363    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv" event={"ID":"7d11ff62-62d9-4300-85aa-d59d3741d587","Type":"ContainerDied","Data":"660e4f9bb696c25978e3ca68017b7ebb33ef5ed7db3cf2e05b324fc1e9b7de88"}
Sep 05 02:54:06.550858 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:06.550431    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv" event={"ID":"7d11ff62-62d9-4300-85aa-d59d3741d587","Type":"ContainerDied","Data":"96689246a64c6c4b3dd96b4d1dc2b54398d2b266b5c970f2ef4477db948e4ff7"}
Sep 05 02:54:06.550858 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:06.550484    2557 scope.go:117] "RemoveContainer" containerID="660e4f9bb696c25978e3ca68017b7ebb33ef5ed7db3cf2e05b324fc1e9b7de88"
Sep 05 02:54:06.560001 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:06.559983    2557 scope.go:117] "RemoveContainer" containerID="de5a5fe1ac008bebe213b934164e9f84ed0209bbbc132aa1cf34c9fb57185d12"
Sep 05 02:54:06.573426 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:06.573406    2557 scope.go:117] "RemoveContainer" containerID="660e4f9bb696c25978e3ca68017b7ebb33ef5ed7db3cf2e05b324fc1e9b7de88"
Sep 05 02:54:06.573787 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:54:06.573765    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660e4f9bb696c25978e3ca68017b7ebb33ef5ed7db3cf2e05b324fc1e9b7de88\": container with ID starting with 660e4f9bb696c25978e3ca68017b7ebb33ef5ed7db3cf2e05b324fc1e9b7de88 not found: ID does not exist" containerID="660e4f9bb696c25978e3ca68017b7ebb33ef5ed7db3cf2e05b324fc1e9b7de88"
Sep 05 02:54:06.573887 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:06.573794    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660e4f9bb696c25978e3ca68017b7ebb33ef5ed7db3cf2e05b324fc1e9b7de88"} err="failed to get container status \"660e4f9bb696c25978e3ca68017b7ebb33ef5ed7db3cf2e05b324fc1e9b7de88\": rpc error: code = NotFound desc = could not find container \"660e4f9bb696c25978e3ca68017b7ebb33ef5ed7db3cf2e05b324fc1e9b7de88\": container with ID starting with 660e4f9bb696c25978e3ca68017b7ebb33ef5ed7db3cf2e05b324fc1e9b7de88 not found: ID does not exist"
Sep 05 02:54:06.573887 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:06.573811    2557 scope.go:117] "RemoveContainer" containerID="de5a5fe1ac008bebe213b934164e9f84ed0209bbbc132aa1cf34c9fb57185d12"
Sep 05 02:54:06.574109 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:54:06.574084    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5a5fe1ac008bebe213b934164e9f84ed0209bbbc132aa1cf34c9fb57185d12\": container with ID starting with de5a5fe1ac008bebe213b934164e9f84ed0209bbbc132aa1cf34c9fb57185d12 not found: ID does not exist" containerID="de5a5fe1ac008bebe213b934164e9f84ed0209bbbc132aa1cf34c9fb57185d12"
Sep 05 02:54:06.574145 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:06.574117    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5a5fe1ac008bebe213b934164e9f84ed0209bbbc132aa1cf34c9fb57185d12"} err="failed to get container status \"de5a5fe1ac008bebe213b934164e9f84ed0209bbbc132aa1cf34c9fb57185d12\": rpc error: code = NotFound desc = could not find container \"de5a5fe1ac008bebe213b934164e9f84ed0209bbbc132aa1cf34c9fb57185d12\": container with ID starting with de5a5fe1ac008bebe213b934164e9f84ed0209bbbc132aa1cf34c9fb57185d12 not found: ID does not exist"
Sep 05 02:54:06.585843 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:06.585810    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"]
Sep 05 02:54:06.592549 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:06.592528    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-786d844b84-9r6xv"]
Sep 05 02:54:06.991135 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:06.991102    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d11ff62-62d9-4300-85aa-d59d3741d587" path="/var/lib/kubelet/pods/7d11ff62-62d9-4300-85aa-d59d3741d587/volumes"
Sep 05 02:54:06.991616 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:06.991600    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" path="/var/lib/kubelet/pods/c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b/volumes"
Sep 05 02:54:10.956838 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:10.956745    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:11.081872 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:11.081827    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:11.179281 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:11.179244    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:11.910680 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:11.910650    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:16.032528 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:16.032493    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:19.640521 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:19.640486    2557 scope.go:117] "RemoveContainer" containerID="83f9057a86303ada2df4a35ca4e7f34546a5d8b1922bd67dbeceee356126fbb9"
Sep 05 02:54:32.598724 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.598691    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"]
Sep 05 02:54:32.599304 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.599230    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a948aff3-a857-4abc-8197-9e48b01043eb" containerName="main"
Sep 05 02:54:32.599304 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.599251    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a948aff3-a857-4abc-8197-9e48b01043eb" containerName="main"
Sep 05 02:54:32.599304 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.599281    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d11ff62-62d9-4300-85aa-d59d3741d587" containerName="storage-initializer"
Sep 05 02:54:32.599304 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.599290    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d11ff62-62d9-4300-85aa-d59d3741d587" containerName="storage-initializer"
Sep 05 02:54:32.599304 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.599301    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" containerName="main"
Sep 05 02:54:32.599608 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.599310    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" containerName="main"
Sep 05 02:54:32.599608 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.599324    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d11ff62-62d9-4300-85aa-d59d3741d587" containerName="main"
Sep 05 02:54:32.599608 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.599331    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d11ff62-62d9-4300-85aa-d59d3741d587" containerName="main"
Sep 05 02:54:32.599608 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.599348    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" containerName="storage-initializer"
Sep 05 02:54:32.599608 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.599356    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" containerName="storage-initializer"
Sep 05 02:54:32.599608 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.599478    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d11ff62-62d9-4300-85aa-d59d3741d587" containerName="main"
Sep 05 02:54:32.599608 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.599492    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a948aff3-a857-4abc-8197-9e48b01043eb" containerName="main"
Sep 05 02:54:32.599608 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.599506    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c85de0cf-2cd7-45ae-8dfa-317fd7bfc07b" containerName="main"
Sep 05 02:54:32.602957 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.602933    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.605953 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.605929    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-r7plc\""
Sep 05 02:54:32.606122 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.605991    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 05 02:54:32.606247 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.605991    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 05 02:54:32.606247 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.606070    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 05 02:54:32.615061 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.615033    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"]
Sep 05 02:54:32.635631 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.635587    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:32.681133 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.681104    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnzdg\" (UniqueName: \"kubernetes.io/projected/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-kube-api-access-tnzdg\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.681296 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.681172    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.681296 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.681232    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.681395 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.681302    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-home\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.681395 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.681343    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.681395 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.681361    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-dshm\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.782310 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.782266    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-home\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.782539 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.782328    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.782539 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.782347    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-dshm\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.782539 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.782460    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnzdg\" (UniqueName: \"kubernetes.io/projected/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-kube-api-access-tnzdg\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.782716 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.782548    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.782716 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.782621    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.782817 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.782750    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.782817 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.782803    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-home\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.782934 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.782915    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.784695 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.784672    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-dshm\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.784946 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.784927    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.790421 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.790401    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnzdg\" (UniqueName: \"kubernetes.io/projected/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-kube-api-access-tnzdg\") pod \"tri-v1-kserve-79f8f98bc5-2gqzw\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:32.915921 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:32.915827    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:33.190789 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:33.188965    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"]
Sep 05 02:54:33.191593 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:54:33.191554    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b8f113_83f3_45e2_a6ac_fee5026d4ea2.slice/crio-28cb74a570dcf962dd9f1fe13cb815122a507f24bed6ad0e4043d3ab30e2b22a WatchSource:0}: Error finding container 28cb74a570dcf962dd9f1fe13cb815122a507f24bed6ad0e4043d3ab30e2b22a: Status 404 returned error can't find the container with id 28cb74a570dcf962dd9f1fe13cb815122a507f24bed6ad0e4043d3ab30e2b22a
Sep 05 02:54:33.696716 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:33.696676    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw" event={"ID":"37b8f113-83f3-45e2-a6ac-fee5026d4ea2","Type":"ContainerStarted","Data":"5badd9ecf12818319bbe022d84b0727e7fbfaebdb70029f11a90ddd8bfab1787"}
Sep 05 02:54:33.697102 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:33.696732    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw" event={"ID":"37b8f113-83f3-45e2-a6ac-fee5026d4ea2","Type":"ContainerStarted","Data":"28cb74a570dcf962dd9f1fe13cb815122a507f24bed6ad0e4043d3ab30e2b22a"}
Sep 05 02:54:33.717936 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:33.717889    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw" podStartSLOduration=1.717875402 podStartE2EDuration="1.717875402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:54:33.714874296 +0000 UTC m=+2177.270030603" watchObservedRunningTime="2026-09-05 02:54:33.717875402 +0000 UTC m=+2177.273031710"
Sep 05 02:54:33.916586 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:33.916551    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:40.603074 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.603026    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"]
Sep 05 02:54:40.607313 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.607288    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.609621 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.609591    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 05 02:54:40.614544 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.614522    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"]
Sep 05 02:54:40.633366 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.633328    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:40.752894 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.752859    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-home\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.752894 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.752894    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-dshm\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.753154 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.752917    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstwt\" (UniqueName: \"kubernetes.io/projected/93ce2c66-820f-4486-862f-d0d3f24aabd8-kube-api-access-xstwt\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.753154 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.752939    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.753154 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.753054    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ce2c66-820f-4486-862f-d0d3f24aabd8-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.753154 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.753098    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-model-cache\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.854405 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.854318    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-dshm\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.854405 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.854356    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstwt\" (UniqueName: \"kubernetes.io/projected/93ce2c66-820f-4486-862f-d0d3f24aabd8-kube-api-access-xstwt\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.854405 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.854380    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.854731 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.854460    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ce2c66-820f-4486-862f-d0d3f24aabd8-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.854731 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.854492    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-model-cache\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.854731 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.854524    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-home\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.854888 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.854841    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.854937 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.854899    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-model-cache\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.854985 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.854969    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-home\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.856663 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.856637    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-dshm\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.857333 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.857313    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ce2c66-820f-4486-862f-d0d3f24aabd8-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.862843 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.862819    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstwt\" (UniqueName: \"kubernetes.io/projected/93ce2c66-820f-4486-862f-d0d3f24aabd8-kube-api-access-xstwt\") pod \"tri-v2-kserve-dcb554bf5-56w75\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:40.920518 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:40.920482    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:41.196245 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:41.196215    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"]
Sep 05 02:54:41.197985 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:54:41.197953    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ce2c66_820f_4486_862f_d0d3f24aabd8.slice/crio-9e9eaa51cad261ec1aac9a9ed7587d78406ae8d7d41be0be524f67d23dc180df WatchSource:0}: Error finding container 9e9eaa51cad261ec1aac9a9ed7587d78406ae8d7d41be0be524f67d23dc180df: Status 404 returned error can't find the container with id 9e9eaa51cad261ec1aac9a9ed7587d78406ae8d7d41be0be524f67d23dc180df
Sep 05 02:54:41.739292 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:41.739250    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75" event={"ID":"93ce2c66-820f-4486-862f-d0d3f24aabd8","Type":"ContainerStarted","Data":"b77c7d4b9e38f782d7c746d4712553cfda6b7811874abfd2658c48142ef6c399"}
Sep 05 02:54:41.739805 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:41.739320    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75" event={"ID":"93ce2c66-820f-4486-862f-d0d3f24aabd8","Type":"ContainerStarted","Data":"9e9eaa51cad261ec1aac9a9ed7587d78406ae8d7d41be0be524f67d23dc180df"}
Sep 05 02:54:41.760102 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:41.760053    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75" podStartSLOduration=1.7600376089999998 podStartE2EDuration="1.760037609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:54:41.758574333 +0000 UTC m=+2185.313730641" watchObservedRunningTime="2026-09-05 02:54:41.760037609 +0000 UTC m=+2185.315193916"
Sep 05 02:54:41.921284 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:41.921242    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:42.916140 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:42.916103    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:42.928578 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:42.928547    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:43.763090 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:43.763049    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:54:49.690686 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.690644    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"]
Sep 05 02:54:49.694429 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.694404    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.696557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.696533    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 05 02:54:49.703647 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.703620    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"]
Sep 05 02:54:49.715388 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.715321    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:49.840303 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.840265    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-tls-certs\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.840303 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.840299    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9vfw\" (UniqueName: \"kubernetes.io/projected/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-kube-api-access-m9vfw\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.840579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.840326    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-dshm\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.840579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.840395    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-home\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.840579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.840458    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-model-cache\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.840579 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.840479    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.942041 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.941923    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-tls-certs\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.942041 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.941983    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9vfw\" (UniqueName: \"kubernetes.io/projected/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-kube-api-access-m9vfw\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.942041 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.942025    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-dshm\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.942318 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.942063    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-home\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.942318 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.942117    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-model-cache\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.942318 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.942145    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.942688 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.942656    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-home\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.942917 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.942898    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.942994 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.942970    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-model-cache\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.944388 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.944370    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-dshm\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.944846 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.944828    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-tls-certs\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:49.950528 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:49.950508    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9vfw\" (UniqueName: \"kubernetes.io/projected/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-kube-api-access-m9vfw\") pod \"tri-v3-kserve-798c88c66c-dqnlh\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:50.006587 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:50.006551    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:50.278014 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:50.277986    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"]
Sep 05 02:54:50.279906 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:54:50.279875    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5bf0b5_954a_4131_b4b7_c5606aa4c8bc.slice/crio-6186df36a73c542b9d97b1d095e5f988c62b7ad78b12640160d7fd9c4e8b9c7d WatchSource:0}: Error finding container 6186df36a73c542b9d97b1d095e5f988c62b7ad78b12640160d7fd9c4e8b9c7d: Status 404 returned error can't find the container with id 6186df36a73c542b9d97b1d095e5f988c62b7ad78b12640160d7fd9c4e8b9c7d
Sep 05 02:54:50.791628 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:50.791593    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh" event={"ID":"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc","Type":"ContainerStarted","Data":"14583a0a90574eea1a03acca6521da9a196c89f6481aae1a72c873d8fa5d3a1c"}
Sep 05 02:54:50.792068 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:50.791661    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh" event={"ID":"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc","Type":"ContainerStarted","Data":"6186df36a73c542b9d97b1d095e5f988c62b7ad78b12640160d7fd9c4e8b9c7d"}
Sep 05 02:54:50.815961 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:50.815831    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh" podStartSLOduration=1.815767058 podStartE2EDuration="1.815767058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:54:50.811933789 +0000 UTC m=+2194.367090100" watchObservedRunningTime="2026-09-05 02:54:50.815767058 +0000 UTC m=+2194.370923367"
Sep 05 02:54:50.828867 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:50.828829    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:50.920990 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:50.920955    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:50.933618 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:50.933580    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:51.007166 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:51.007130    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:54:51.807950 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:51.807921    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:54:53.540392 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:53.540358    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:54.925315 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:54.925280    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:55.024837 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:55.024800    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:59.117721 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:59.117685    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:54:59.182278 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:54:59.182231    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:55:00.007015 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:00.006977    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:55:00.019053 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:00.019024    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:55:00.855823 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:00.855790    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:55:01.220728 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:01.220642    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:55:06.779410 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:06.779363    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:55:06.878369 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:06.878333    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:55:14.928605 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:14.928572    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"]
Sep 05 02:55:14.932664 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:14.932631    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:14.935667 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:14.935642    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-9q65n\""
Sep 05 02:55:14.935817 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:14.935642    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 05 02:55:14.935817 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:14.935651    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 05 02:55:14.935817 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:14.935717    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 05 02:55:14.935817 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:14.935655    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:55:14.935817 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:14.935717    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 05 02:55:14.943761 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:14.943736    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"]
Sep 05 02:55:15.006876 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.006837    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:55:15.083628 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.083586    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.083628 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.083634    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-model-cache\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.083888 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.083694    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.083888 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.083752    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9ab8327-3443-421b-8f9e-4554e631ba26-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.083888 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.083857    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ab8327-3443-421b-8f9e-4554e631ba26-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.084041 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.083923    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-dshm\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.084041 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.083955    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-home\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.084041 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.083980    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbp7w\" (UniqueName: \"kubernetes.io/projected/c9ab8327-3443-421b-8f9e-4554e631ba26-kube-api-access-lbp7w\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.185103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.185007    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-dshm\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.185103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.185049    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-home\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.185103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.185070    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbp7w\" (UniqueName: \"kubernetes.io/projected/c9ab8327-3443-421b-8f9e-4554e631ba26-kube-api-access-lbp7w\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.185103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.185093    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.185459 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.185124    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-model-cache\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.185459 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.185178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.185459 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.185252    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9ab8327-3443-421b-8f9e-4554e631ba26-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.185459 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.185305    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ab8327-3443-421b-8f9e-4554e631ba26-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.185677 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.185587    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-model-cache\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.185677 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.185616    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-home\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.185677 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.185645    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.185969 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.185942    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.186325 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.186277    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9ab8327-3443-421b-8f9e-4554e631ba26-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.187561 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.187526    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-dshm\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.187811 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.187787    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ab8327-3443-421b-8f9e-4554e631ba26-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.194138 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.194116    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbp7w\" (UniqueName: \"kubernetes.io/projected/c9ab8327-3443-421b-8f9e-4554e631ba26-kube-api-access-lbp7w\") pod \"tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.209371 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.209341    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"]
Sep 05 02:55:15.214163 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.214142    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:15.222325 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.222297    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"]
Sep 05 02:55:15.245418 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.245389    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:15.286237 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.286199    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:15.286418 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.286245    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:15.286418 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.286268    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:15.286564 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.286459    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:15.387770 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.387733    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:15.387969 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.387801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:15.387969 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.387824    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:15.387969 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.387850    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:15.388283 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.388258    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:15.388346 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.388326    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:15.388542 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.388526    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:15.390311 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.390290    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:15.497279 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.497193    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:55:15.526173 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.526152    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:15.526986 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.526964    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"]
Sep 05 02:55:15.528231 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:55:15.528205    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ab8327_3443_421b_8f9e_4554e631ba26.slice/crio-aa53d0f54875af0eb777f63ca8caf9ff52659534b0e796ff577e5f8bce9bba84 WatchSource:0}: Error finding container aa53d0f54875af0eb777f63ca8caf9ff52659534b0e796ff577e5f8bce9bba84: Status 404 returned error can't find the container with id aa53d0f54875af0eb777f63ca8caf9ff52659534b0e796ff577e5f8bce9bba84
Sep 05 02:55:15.597304 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.597203    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:55:15.814167 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.814142    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"]
Sep 05 02:55:15.816366 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:55:15.816339    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1e66ef_fc83_49f3_b049_eb345cbf9eec.slice/crio-17c1fa3a3021908b476c1b6ffda2b6dbc6349a055036e92588c287ed3d6ff2af WatchSource:0}: Error finding container 17c1fa3a3021908b476c1b6ffda2b6dbc6349a055036e92588c287ed3d6ff2af: Status 404 returned error can't find the container with id 17c1fa3a3021908b476c1b6ffda2b6dbc6349a055036e92588c287ed3d6ff2af
Sep 05 02:55:15.934199 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.934159    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9" event={"ID":"5c1e66ef-fc83-49f3-b049-eb345cbf9eec","Type":"ContainerStarted","Data":"3d2a1a1a311e251a42624fb66cdc4ab30365473cd174263b222c4013ffa2c444"}
Sep 05 02:55:15.934664 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.934221    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9" event={"ID":"5c1e66ef-fc83-49f3-b049-eb345cbf9eec","Type":"ContainerStarted","Data":"17c1fa3a3021908b476c1b6ffda2b6dbc6349a055036e92588c287ed3d6ff2af"}
Sep 05 02:55:15.942974 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.942936    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np" event={"ID":"c9ab8327-3443-421b-8f9e-4554e631ba26","Type":"ContainerStarted","Data":"ea00e809c953f8951797e00c8da76927a809a94fa6bc75f8226f823db29fe1b3"}
Sep 05 02:55:15.943121 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:15.943042    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np" event={"ID":"c9ab8327-3443-421b-8f9e-4554e631ba26","Type":"ContainerStarted","Data":"aa53d0f54875af0eb777f63ca8caf9ff52659534b0e796ff577e5f8bce9bba84"}
Sep 05 02:55:18.967455 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:18.967399    2557 generic.go:356] "Generic (PLEG): container finished" podID="5c1e66ef-fc83-49f3-b049-eb345cbf9eec" containerID="3d2a1a1a311e251a42624fb66cdc4ab30365473cd174263b222c4013ffa2c444" exitCode=0
Sep 05 02:55:18.967959 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:18.967480    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9" event={"ID":"5c1e66ef-fc83-49f3-b049-eb345cbf9eec","Type":"ContainerDied","Data":"3d2a1a1a311e251a42624fb66cdc4ab30365473cd174263b222c4013ffa2c444"}
Sep 05 02:55:18.970035 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:18.970004    2557 generic.go:356] "Generic (PLEG): container finished" podID="c9ab8327-3443-421b-8f9e-4554e631ba26" containerID="ea00e809c953f8951797e00c8da76927a809a94fa6bc75f8226f823db29fe1b3" exitCode=0
Sep 05 02:55:18.970154 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:18.970076    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np" event={"ID":"c9ab8327-3443-421b-8f9e-4554e631ba26","Type":"ContainerDied","Data":"ea00e809c953f8951797e00c8da76927a809a94fa6bc75f8226f823db29fe1b3"}
Sep 05 02:55:19.978427 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:19.978386    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9" event={"ID":"5c1e66ef-fc83-49f3-b049-eb345cbf9eec","Type":"ContainerStarted","Data":"88d2ddfb669c865fa3bc208180a988d06c6bb1465997384a7ff61d0a7f7d8c5c"}
Sep 05 02:55:19.978874 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:19.978660    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:19.980524 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:19.980502    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np" event={"ID":"c9ab8327-3443-421b-8f9e-4554e631ba26","Type":"ContainerStarted","Data":"e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881"}
Sep 05 02:55:19.998105 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:19.998057    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9" podStartSLOduration=2.509539738 podStartE2EDuration="4.998044629s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.488504891s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:55:19.996177064 +0000 UTC m=+2223.551333386" watchObservedRunningTime="2026-09-05 02:55:19.998044629 +0000 UTC m=+2223.553200937"
Sep 05 02:55:20.020122 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:20.020070    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np" podStartSLOduration=3.52164098 podStartE2EDuration="6.020058247s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.498417267s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:55:20.017245812 +0000 UTC m=+2223.572402133" watchObservedRunningTime="2026-09-05 02:55:20.020058247 +0000 UTC m=+2223.575214600"
Sep 05 02:55:20.246321 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:20.246224    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:25.246259 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:25.246226    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:25.259187 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:25.259150    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:26.031746 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:26.031706    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:26.907825 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:26.907782    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:55:50.997338 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:50.997261    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:52.454223 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:52.454187    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9_5c1e66ef-fc83-49f3-b049-eb345cbf9eec/main/0.log"
Sep 05 02:55:52.630157 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:52.630119    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"]
Sep 05 02:55:52.633821 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:52.633788    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"]
Sep 05 02:55:52.634005 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:52.633983    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9" podUID="5c1e66ef-fc83-49f3-b049-eb345cbf9eec" containerName="main" containerID="cri-o://88d2ddfb669c865fa3bc208180a988d06c6bb1465997384a7ff61d0a7f7d8c5c" gracePeriod=30
Sep 05 02:55:52.640942 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:55:52.640898    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:55:52.640942 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:55:52.640942 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:55:52.640942 ip-10-0-143-128 kubenswrapper[2557]:  > containerID="e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881" cmd=["/bin/sleep","15"]
Sep 05 02:55:52.641153 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:52.640964    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np" message="" err=<
Sep 05 02:55:52.641153 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:55:52.641153 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:55:52.641153 ip-10-0-143-128 kubenswrapper[2557]:  >
Sep 05 02:55:52.641153 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:55:52.640983    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:55:52.641153 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:55:52.641153 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:55:52.641153 ip-10-0-143-128 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np" podUID="c9ab8327-3443-421b-8f9e-4554e631ba26" containerName="main" containerID="cri-o://e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881"
Sep 05 02:55:52.641153 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:52.641026    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np" podUID="c9ab8327-3443-421b-8f9e-4554e631ba26" containerName="main" containerID="cri-o://e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881" gracePeriod=60
Sep 05 02:55:52.685643 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:52.685608    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:55:53.000788 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.000712    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:53.154771 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.153747    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-kserve-provision-location\") pod \"c9ab8327-3443-421b-8f9e-4554e631ba26\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") "
Sep 05 02:55:53.154771 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.153862    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9ab8327-3443-421b-8f9e-4554e631ba26-cabundle-cert\") pod \"c9ab8327-3443-421b-8f9e-4554e631ba26\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") "
Sep 05 02:55:53.154771 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.153917    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ab8327-3443-421b-8f9e-4554e631ba26-tls-certs\") pod \"c9ab8327-3443-421b-8f9e-4554e631ba26\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") "
Sep 05 02:55:53.154771 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.153942    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-home\") pod \"c9ab8327-3443-421b-8f9e-4554e631ba26\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") "
Sep 05 02:55:53.154771 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.153978    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbp7w\" (UniqueName: \"kubernetes.io/projected/c9ab8327-3443-421b-8f9e-4554e631ba26-kube-api-access-lbp7w\") pod \"c9ab8327-3443-421b-8f9e-4554e631ba26\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") "
Sep 05 02:55:53.154771 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.154017    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-model-cache\") pod \"c9ab8327-3443-421b-8f9e-4554e631ba26\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") "
Sep 05 02:55:53.154771 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.154045    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-tmp-dir\") pod \"c9ab8327-3443-421b-8f9e-4554e631ba26\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") "
Sep 05 02:55:53.154771 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.154070    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-dshm\") pod \"c9ab8327-3443-421b-8f9e-4554e631ba26\" (UID: \"c9ab8327-3443-421b-8f9e-4554e631ba26\") "
Sep 05 02:55:53.155381 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.155143    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-home" (OuterVolumeSpecName: "home") pod "c9ab8327-3443-421b-8f9e-4554e631ba26" (UID: "c9ab8327-3443-421b-8f9e-4554e631ba26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:53.155753 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.155693    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9ab8327-3443-421b-8f9e-4554e631ba26" (UID: "c9ab8327-3443-421b-8f9e-4554e631ba26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:53.155859 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.155747    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ab8327-3443-421b-8f9e-4554e631ba26-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c9ab8327-3443-421b-8f9e-4554e631ba26" (UID: "c9ab8327-3443-421b-8f9e-4554e631ba26"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:55:53.156403 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.156379    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9ab8327-3443-421b-8f9e-4554e631ba26" (UID: "c9ab8327-3443-421b-8f9e-4554e631ba26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:53.157106 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.157068    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-dshm" (OuterVolumeSpecName: "dshm") pod "c9ab8327-3443-421b-8f9e-4554e631ba26" (UID: "c9ab8327-3443-421b-8f9e-4554e631ba26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:53.161178 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.161150    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ab8327-3443-421b-8f9e-4554e631ba26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9ab8327-3443-421b-8f9e-4554e631ba26" (UID: "c9ab8327-3443-421b-8f9e-4554e631ba26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:55:53.174638 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.174606    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ab8327-3443-421b-8f9e-4554e631ba26-kube-api-access-lbp7w" (OuterVolumeSpecName: "kube-api-access-lbp7w") pod "c9ab8327-3443-421b-8f9e-4554e631ba26" (UID: "c9ab8327-3443-421b-8f9e-4554e631ba26"). InnerVolumeSpecName "kube-api-access-lbp7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:55:53.181945 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.181752    2557 generic.go:356] "Generic (PLEG): container finished" podID="5c1e66ef-fc83-49f3-b049-eb345cbf9eec" containerID="88d2ddfb669c865fa3bc208180a988d06c6bb1465997384a7ff61d0a7f7d8c5c" exitCode=0
Sep 05 02:55:53.181945 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.181837    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9" event={"ID":"5c1e66ef-fc83-49f3-b049-eb345cbf9eec","Type":"ContainerDied","Data":"88d2ddfb669c865fa3bc208180a988d06c6bb1465997384a7ff61d0a7f7d8c5c"}
Sep 05 02:55:53.184189 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.184153    2557 generic.go:356] "Generic (PLEG): container finished" podID="c9ab8327-3443-421b-8f9e-4554e631ba26" containerID="e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881" exitCode=0
Sep 05 02:55:53.184320 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.184211    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"
Sep 05 02:55:53.184320 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.184330    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np" event={"ID":"c9ab8327-3443-421b-8f9e-4554e631ba26","Type":"ContainerDied","Data":"e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881"}
Sep 05 02:55:53.184565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.184378    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np" event={"ID":"c9ab8327-3443-421b-8f9e-4554e631ba26","Type":"ContainerDied","Data":"aa53d0f54875af0eb777f63ca8caf9ff52659534b0e796ff577e5f8bce9bba84"}
Sep 05 02:55:53.184565 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.184413    2557 scope.go:117] "RemoveContainer" containerID="e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881"
Sep 05 02:55:53.190892 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.190864    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c9ab8327-3443-421b-8f9e-4554e631ba26" (UID: "c9ab8327-3443-421b-8f9e-4554e631ba26"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:53.196710 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.196693    2557 scope.go:117] "RemoveContainer" containerID="ea00e809c953f8951797e00c8da76927a809a94fa6bc75f8226f823db29fe1b3"
Sep 05 02:55:53.209115 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.209091    2557 scope.go:117] "RemoveContainer" containerID="e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881"
Sep 05 02:55:53.209457 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:55:53.209415    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881\": container with ID starting with e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881 not found: ID does not exist" containerID="e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881"
Sep 05 02:55:53.209574 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.209464    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881"} err="failed to get container status \"e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881\": rpc error: code = NotFound desc = could not find container \"e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881\": container with ID starting with e940f8852d131f052ec6a441e23e749943658bd26247bafdb602aa56ba0e8881 not found: ID does not exist"
Sep 05 02:55:53.209574 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.209489    2557 scope.go:117] "RemoveContainer" containerID="ea00e809c953f8951797e00c8da76927a809a94fa6bc75f8226f823db29fe1b3"
Sep 05 02:55:53.209810 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:55:53.209778    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea00e809c953f8951797e00c8da76927a809a94fa6bc75f8226f823db29fe1b3\": container with ID starting with ea00e809c953f8951797e00c8da76927a809a94fa6bc75f8226f823db29fe1b3 not found: ID does not exist" containerID="ea00e809c953f8951797e00c8da76927a809a94fa6bc75f8226f823db29fe1b3"
Sep 05 02:55:53.209878 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.209814    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea00e809c953f8951797e00c8da76927a809a94fa6bc75f8226f823db29fe1b3"} err="failed to get container status \"ea00e809c953f8951797e00c8da76927a809a94fa6bc75f8226f823db29fe1b3\": rpc error: code = NotFound desc = could not find container \"ea00e809c953f8951797e00c8da76927a809a94fa6bc75f8226f823db29fe1b3\": container with ID starting with ea00e809c953f8951797e00c8da76927a809a94fa6bc75f8226f823db29fe1b3 not found: ID does not exist"
Sep 05 02:55:53.256137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.256056    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9ab8327-3443-421b-8f9e-4554e631ba26-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.256137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.256098    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ab8327-3443-421b-8f9e-4554e631ba26-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.256137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.256112    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.256137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.256125    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbp7w\" (UniqueName: \"kubernetes.io/projected/c9ab8327-3443-421b-8f9e-4554e631ba26-kube-api-access-lbp7w\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.256137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.256139    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.256561 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.256152    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.256561 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.256164    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.256561 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.256176    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9ab8327-3443-421b-8f9e-4554e631ba26-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.304560 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.304532    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:53.457257 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.457221    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-tls-certs\") pod \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") "
Sep 05 02:55:53.457747 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.457266    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-cabundle-cert\") pod \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") "
Sep 05 02:55:53.457747 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.457301    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-kserve-provision-location\") pod \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") "
Sep 05 02:55:53.457747 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.457376    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-tmp-dir\") pod \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\" (UID: \"5c1e66ef-fc83-49f3-b049-eb345cbf9eec\") "
Sep 05 02:55:53.457946 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.457919    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c1e66ef-fc83-49f3-b049-eb345cbf9eec" (UID: "5c1e66ef-fc83-49f3-b049-eb345cbf9eec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:55:53.458016 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.457965    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c1e66ef-fc83-49f3-b049-eb345cbf9eec" (UID: "5c1e66ef-fc83-49f3-b049-eb345cbf9eec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:53.459421 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.459399    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c1e66ef-fc83-49f3-b049-eb345cbf9eec" (UID: "5c1e66ef-fc83-49f3-b049-eb345cbf9eec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:55:53.480286 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.480256    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c1e66ef-fc83-49f3-b049-eb345cbf9eec" (UID: "5c1e66ef-fc83-49f3-b049-eb345cbf9eec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:55:53.506583 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.506525    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"]
Sep 05 02:55:53.510057 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.510035    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7c5b679cfb-vd2np"]
Sep 05 02:55:53.558556 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.558526    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.558556 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.558555    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.558735 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.558565    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:55:53.558735 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:53.558574    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c1e66ef-fc83-49f3-b049-eb345cbf9eec-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:55:54.190672 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:54.190635    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9" event={"ID":"5c1e66ef-fc83-49f3-b049-eb345cbf9eec","Type":"ContainerDied","Data":"17c1fa3a3021908b476c1b6ffda2b6dbc6349a055036e92588c287ed3d6ff2af"}
Sep 05 02:55:54.190672 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:54.190659    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"
Sep 05 02:55:54.190939 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:54.190709    2557 scope.go:117] "RemoveContainer" containerID="88d2ddfb669c865fa3bc208180a988d06c6bb1465997384a7ff61d0a7f7d8c5c"
Sep 05 02:55:54.199690 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:54.199668    2557 scope.go:117] "RemoveContainer" containerID="3d2a1a1a311e251a42624fb66cdc4ab30365473cd174263b222c4013ffa2c444"
Sep 05 02:55:54.212876 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:54.212853    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"]
Sep 05 02:55:54.216533 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:54.216509    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-bffcf5d-bmls9"]
Sep 05 02:55:54.883316 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:54.883276    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:55:54.990881 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:54.990850    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c1e66ef-fc83-49f3-b049-eb345cbf9eec" path="/var/lib/kubelet/pods/5c1e66ef-fc83-49f3-b049-eb345cbf9eec/volumes"
Sep 05 02:55:54.991332 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:55:54.991316    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9ab8327-3443-421b-8f9e-4554e631ba26" path="/var/lib/kubelet/pods/c9ab8327-3443-421b-8f9e-4554e631ba26/volumes"
Sep 05 02:56:04.137814 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:04.137777    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:04.238979 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:04.238940    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:04.341298 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:04.341257    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:04.796349 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:04.796316    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"]
Sep 05 02:56:04.804907 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:04.804875    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:05.396596 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:05.396555    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"]
Sep 05 02:56:05.424951 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:05.424917    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:05.998351 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:05.998319    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"]
Sep 05 02:56:06.021956 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:06.021905    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:09.212571 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:09.212536    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:16.212621 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.212584    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"]
Sep 05 02:56:16.212996 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.212983    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c1e66ef-fc83-49f3-b049-eb345cbf9eec" containerName="main"
Sep 05 02:56:16.212996 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.212996    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c1e66ef-fc83-49f3-b049-eb345cbf9eec" containerName="main"
Sep 05 02:56:16.213070 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.213009    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9ab8327-3443-421b-8f9e-4554e631ba26" containerName="main"
Sep 05 02:56:16.213070 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.213016    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9ab8327-3443-421b-8f9e-4554e631ba26" containerName="main"
Sep 05 02:56:16.213070 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.213029    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9ab8327-3443-421b-8f9e-4554e631ba26" containerName="storage-initializer"
Sep 05 02:56:16.213070 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.213035    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9ab8327-3443-421b-8f9e-4554e631ba26" containerName="storage-initializer"
Sep 05 02:56:16.213070 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.213042    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c1e66ef-fc83-49f3-b049-eb345cbf9eec" containerName="storage-initializer"
Sep 05 02:56:16.213277 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.213049    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c1e66ef-fc83-49f3-b049-eb345cbf9eec" containerName="storage-initializer"
Sep 05 02:56:16.213277 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.213178    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9ab8327-3443-421b-8f9e-4554e631ba26" containerName="main"
Sep 05 02:56:16.213277 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.213188    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c1e66ef-fc83-49f3-b049-eb345cbf9eec" containerName="main"
Sep 05 02:56:16.216846 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.216821    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.220131 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.220104    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-jtnc9\""
Sep 05 02:56:16.220131 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.220120    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 05 02:56:16.220333 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.220123    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 05 02:56:16.220333 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.220115    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 05 02:56:16.226093 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.226068    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"]
Sep 05 02:56:16.239656 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.239605    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:16.270822 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.270789    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-home\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.270978 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.270829    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x58l5\" (UniqueName: \"kubernetes.io/projected/0fb8e043-2935-4e19-b01a-ccaf34c43046-kube-api-access-x58l5\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.270978 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.270887    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb8e043-2935-4e19-b01a-ccaf34c43046-tls-certs\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.270978 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.270925    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-dshm\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.271081 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.270973    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.271081 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.271001    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-model-cache\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.371674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.371637    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-home\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.371674 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.371684    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x58l5\" (UniqueName: \"kubernetes.io/projected/0fb8e043-2935-4e19-b01a-ccaf34c43046-kube-api-access-x58l5\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.371902 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.371822    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb8e043-2935-4e19-b01a-ccaf34c43046-tls-certs\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.371902 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.371866    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-dshm\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.371902 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.371890    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.372060 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.371917    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-model-cache\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.372117 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.372099    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-home\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.372308 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.372285    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.372366 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.372298    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-model-cache\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.374198 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.374164    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-dshm\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.374338 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.374322    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb8e043-2935-4e19-b01a-ccaf34c43046-tls-certs\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.379296 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.379272    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x58l5\" (UniqueName: \"kubernetes.io/projected/0fb8e043-2935-4e19-b01a-ccaf34c43046-kube-api-access-x58l5\") pod \"late-v1-kserve-56cbc8456c-csqxl\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.528952 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.528864    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:16.814936 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:16.814907    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"]
Sep 05 02:56:16.816562 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:56:16.816524    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb8e043_2935_4e19_b01a_ccaf34c43046.slice/crio-ceab8c88276df66065cdc4e89990d049ca989f9b9715d174dccc53e07409a6f7 WatchSource:0}: Error finding container ceab8c88276df66065cdc4e89990d049ca989f9b9715d174dccc53e07409a6f7: Status 404 returned error can't find the container with id ceab8c88276df66065cdc4e89990d049ca989f9b9715d174dccc53e07409a6f7
Sep 05 02:56:17.319570 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:17.319530    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl" event={"ID":"0fb8e043-2935-4e19-b01a-ccaf34c43046","Type":"ContainerStarted","Data":"5e62b6a8e52e7b70707ad0cf4366b9d867d3b7bcad23ab0fd99535deb594e8e7"}
Sep 05 02:56:17.319937 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:17.319600    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl" event={"ID":"0fb8e043-2935-4e19-b01a-ccaf34c43046","Type":"ContainerStarted","Data":"ceab8c88276df66065cdc4e89990d049ca989f9b9715d174dccc53e07409a6f7"}
Sep 05 02:56:17.338529 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:17.338466    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl" podStartSLOduration=1.338425095 podStartE2EDuration="1.338425095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:56:17.337535387 +0000 UTC m=+2280.892691696" watchObservedRunningTime="2026-09-05 02:56:17.338425095 +0000 UTC m=+2280.893581416"
Sep 05 02:56:17.529384 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:17.529333    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:19.816123 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:19.816087    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh" podUID="4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc" containerName="main" containerID="cri-o://14583a0a90574eea1a03acca6521da9a196c89f6481aae1a72c873d8fa5d3a1c" gracePeriod=45
Sep 05 02:56:19.845823 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:19.845788    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh" podUID="4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 05 02:56:20.071152 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.071129    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:56:20.101633 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.101601    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-model-cache\") pod \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") "
Sep 05 02:56:20.101770 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.101672    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-tmp-dir\") pod \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") "
Sep 05 02:56:20.101770 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.101724    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-tls-certs\") pod \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") "
Sep 05 02:56:20.101770 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.101742    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-dshm\") pod \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") "
Sep 05 02:56:20.101770 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.101762    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-home\") pod \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") "
Sep 05 02:56:20.101970 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.101815    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9vfw\" (UniqueName: \"kubernetes.io/projected/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-kube-api-access-m9vfw\") pod \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\" (UID: \"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc\") "
Sep 05 02:56:20.101970 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.101891    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc" (UID: "4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:56:20.101970 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.101919    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc" (UID: "4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:56:20.102127 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.102046    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-home" (OuterVolumeSpecName: "home") pod "4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc" (UID: "4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:56:20.102179 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.102166    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:20.102232 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.102184    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:20.102232 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.102198    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:20.103896 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.103872    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-dshm" (OuterVolumeSpecName: "dshm") pod "4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc" (UID: "4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:56:20.104271 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.104248    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-kube-api-access-m9vfw" (OuterVolumeSpecName: "kube-api-access-m9vfw") pod "4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc" (UID: "4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc"). InnerVolumeSpecName "kube-api-access-m9vfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:56:20.104394 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.104254    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc" (UID: "4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:56:20.203515 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.203426    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9vfw\" (UniqueName: \"kubernetes.io/projected/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-kube-api-access-m9vfw\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:20.203515 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.203471    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:20.203515 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.203480    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:20.337233 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.337203    2557 generic.go:356] "Generic (PLEG): container finished" podID="4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc" containerID="14583a0a90574eea1a03acca6521da9a196c89f6481aae1a72c873d8fa5d3a1c" exitCode=0
Sep 05 02:56:20.337395 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.337243    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh" event={"ID":"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc","Type":"ContainerDied","Data":"14583a0a90574eea1a03acca6521da9a196c89f6481aae1a72c873d8fa5d3a1c"}
Sep 05 02:56:20.337395 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.337248    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"
Sep 05 02:56:20.337395 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.337279    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh" event={"ID":"4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc","Type":"ContainerDied","Data":"6186df36a73c542b9d97b1d095e5f988c62b7ad78b12640160d7fd9c4e8b9c7d"}
Sep 05 02:56:20.337395 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.337301    2557 scope.go:117] "RemoveContainer" containerID="14583a0a90574eea1a03acca6521da9a196c89f6481aae1a72c873d8fa5d3a1c"
Sep 05 02:56:20.348948 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.347540    2557 scope.go:117] "RemoveContainer" containerID="14583a0a90574eea1a03acca6521da9a196c89f6481aae1a72c873d8fa5d3a1c"
Sep 05 02:56:20.349257 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:56:20.349235    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14583a0a90574eea1a03acca6521da9a196c89f6481aae1a72c873d8fa5d3a1c\": container with ID starting with 14583a0a90574eea1a03acca6521da9a196c89f6481aae1a72c873d8fa5d3a1c not found: ID does not exist" containerID="14583a0a90574eea1a03acca6521da9a196c89f6481aae1a72c873d8fa5d3a1c"
Sep 05 02:56:20.349311 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.349265    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14583a0a90574eea1a03acca6521da9a196c89f6481aae1a72c873d8fa5d3a1c"} err="failed to get container status \"14583a0a90574eea1a03acca6521da9a196c89f6481aae1a72c873d8fa5d3a1c\": rpc error: code = NotFound desc = could not find container \"14583a0a90574eea1a03acca6521da9a196c89f6481aae1a72c873d8fa5d3a1c\": container with ID starting with 14583a0a90574eea1a03acca6521da9a196c89f6481aae1a72c873d8fa5d3a1c not found: ID does not exist"
Sep 05 02:56:20.359501 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.359479    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"]
Sep 05 02:56:20.363257 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.363238    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dqnlh"]
Sep 05 02:56:20.415496 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.415468    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw" podUID="37b8f113-83f3-45e2-a6ac-fee5026d4ea2" containerName="main" containerID="cri-o://5badd9ecf12818319bbe022d84b0727e7fbfaebdb70029f11a90ddd8bfab1787" gracePeriod=45
Sep 05 02:56:20.650222 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.650197    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:56:20.707151 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.707120    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-tls-certs\") pod \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") "
Sep 05 02:56:20.707314 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.707197    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnzdg\" (UniqueName: \"kubernetes.io/projected/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-kube-api-access-tnzdg\") pod \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") "
Sep 05 02:56:20.707314 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.707225    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-home\") pod \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") "
Sep 05 02:56:20.707314 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.707251    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-tmp-dir\") pod \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") "
Sep 05 02:56:20.707314 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.707273    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-dshm\") pod \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") "
Sep 05 02:56:20.707314 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.707298    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-model-cache\") pod \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\" (UID: \"37b8f113-83f3-45e2-a6ac-fee5026d4ea2\") "
Sep 05 02:56:20.707637 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.707586    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37b8f113-83f3-45e2-a6ac-fee5026d4ea2" (UID: "37b8f113-83f3-45e2-a6ac-fee5026d4ea2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:56:20.707637 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.707597    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-home" (OuterVolumeSpecName: "home") pod "37b8f113-83f3-45e2-a6ac-fee5026d4ea2" (UID: "37b8f113-83f3-45e2-a6ac-fee5026d4ea2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:56:20.707741 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.707636    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-model-cache" (OuterVolumeSpecName: "model-cache") pod "37b8f113-83f3-45e2-a6ac-fee5026d4ea2" (UID: "37b8f113-83f3-45e2-a6ac-fee5026d4ea2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:56:20.709248 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.709217    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37b8f113-83f3-45e2-a6ac-fee5026d4ea2" (UID: "37b8f113-83f3-45e2-a6ac-fee5026d4ea2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:56:20.709367 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.709316    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-dshm" (OuterVolumeSpecName: "dshm") pod "37b8f113-83f3-45e2-a6ac-fee5026d4ea2" (UID: "37b8f113-83f3-45e2-a6ac-fee5026d4ea2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:56:20.709367 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.709343    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-kube-api-access-tnzdg" (OuterVolumeSpecName: "kube-api-access-tnzdg") pod "37b8f113-83f3-45e2-a6ac-fee5026d4ea2" (UID: "37b8f113-83f3-45e2-a6ac-fee5026d4ea2"). InnerVolumeSpecName "kube-api-access-tnzdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:56:20.808315 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.808233    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:20.808315 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.808261    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnzdg\" (UniqueName: \"kubernetes.io/projected/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-kube-api-access-tnzdg\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:20.808315 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.808272    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:20.808315 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.808286    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:20.808315 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.808295    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:20.808315 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.808302    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37b8f113-83f3-45e2-a6ac-fee5026d4ea2-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:20.990482 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:20.990424    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc" path="/var/lib/kubelet/pods/4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc/volumes"
Sep 05 02:56:21.016993 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.016940    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75" podUID="93ce2c66-820f-4486-862f-d0d3f24aabd8" containerName="main" containerID="cri-o://b77c7d4b9e38f782d7c746d4712553cfda6b7811874abfd2658c48142ef6c399" gracePeriod=45
Sep 05 02:56:21.254623 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.254600    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:56:21.312885 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.312856    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-dshm\") pod \"93ce2c66-820f-4486-862f-d0d3f24aabd8\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") "
Sep 05 02:56:21.313022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.312920    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ce2c66-820f-4486-862f-d0d3f24aabd8-tls-certs\") pod \"93ce2c66-820f-4486-862f-d0d3f24aabd8\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") "
Sep 05 02:56:21.313022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.312940    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-model-cache\") pod \"93ce2c66-820f-4486-862f-d0d3f24aabd8\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") "
Sep 05 02:56:21.313022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.312966    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-home\") pod \"93ce2c66-820f-4486-862f-d0d3f24aabd8\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") "
Sep 05 02:56:21.313022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.313006    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xstwt\" (UniqueName: \"kubernetes.io/projected/93ce2c66-820f-4486-862f-d0d3f24aabd8-kube-api-access-xstwt\") pod \"93ce2c66-820f-4486-862f-d0d3f24aabd8\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") "
Sep 05 02:56:21.313220 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.313082    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-tmp-dir\") pod \"93ce2c66-820f-4486-862f-d0d3f24aabd8\" (UID: \"93ce2c66-820f-4486-862f-d0d3f24aabd8\") "
Sep 05 02:56:21.313270 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.313232    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-home" (OuterVolumeSpecName: "home") pod "93ce2c66-820f-4486-862f-d0d3f24aabd8" (UID: "93ce2c66-820f-4486-862f-d0d3f24aabd8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:56:21.313318 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.313244    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-model-cache" (OuterVolumeSpecName: "model-cache") pod "93ce2c66-820f-4486-862f-d0d3f24aabd8" (UID: "93ce2c66-820f-4486-862f-d0d3f24aabd8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:56:21.313360 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.313326    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "93ce2c66-820f-4486-862f-d0d3f24aabd8" (UID: "93ce2c66-820f-4486-862f-d0d3f24aabd8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:56:21.313489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.313471    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:21.313592 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.313490    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:21.313592 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.313502    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:21.314871 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.314843    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-dshm" (OuterVolumeSpecName: "dshm") pod "93ce2c66-820f-4486-862f-d0d3f24aabd8" (UID: "93ce2c66-820f-4486-862f-d0d3f24aabd8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:56:21.314986 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.314948    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ce2c66-820f-4486-862f-d0d3f24aabd8-kube-api-access-xstwt" (OuterVolumeSpecName: "kube-api-access-xstwt") pod "93ce2c66-820f-4486-862f-d0d3f24aabd8" (UID: "93ce2c66-820f-4486-862f-d0d3f24aabd8"). InnerVolumeSpecName "kube-api-access-xstwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:56:21.314986 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.314965    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ce2c66-820f-4486-862f-d0d3f24aabd8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93ce2c66-820f-4486-862f-d0d3f24aabd8" (UID: "93ce2c66-820f-4486-862f-d0d3f24aabd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:56:21.343508 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.343476    2557 generic.go:356] "Generic (PLEG): container finished" podID="93ce2c66-820f-4486-862f-d0d3f24aabd8" containerID="b77c7d4b9e38f782d7c746d4712553cfda6b7811874abfd2658c48142ef6c399" exitCode=0
Sep 05 02:56:21.343627 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.343570    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"
Sep 05 02:56:21.343693 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.343570    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75" event={"ID":"93ce2c66-820f-4486-862f-d0d3f24aabd8","Type":"ContainerDied","Data":"b77c7d4b9e38f782d7c746d4712553cfda6b7811874abfd2658c48142ef6c399"}
Sep 05 02:56:21.343743 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.343727    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75" event={"ID":"93ce2c66-820f-4486-862f-d0d3f24aabd8","Type":"ContainerDied","Data":"9e9eaa51cad261ec1aac9a9ed7587d78406ae8d7d41be0be524f67d23dc180df"}
Sep 05 02:56:21.343792 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.343761    2557 scope.go:117] "RemoveContainer" containerID="b77c7d4b9e38f782d7c746d4712553cfda6b7811874abfd2658c48142ef6c399"
Sep 05 02:56:21.346290 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.346266    2557 generic.go:356] "Generic (PLEG): container finished" podID="37b8f113-83f3-45e2-a6ac-fee5026d4ea2" containerID="5badd9ecf12818319bbe022d84b0727e7fbfaebdb70029f11a90ddd8bfab1787" exitCode=0
Sep 05 02:56:21.346401 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.346310    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw" event={"ID":"37b8f113-83f3-45e2-a6ac-fee5026d4ea2","Type":"ContainerDied","Data":"5badd9ecf12818319bbe022d84b0727e7fbfaebdb70029f11a90ddd8bfab1787"}
Sep 05 02:56:21.346401 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.346366    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw" event={"ID":"37b8f113-83f3-45e2-a6ac-fee5026d4ea2","Type":"ContainerDied","Data":"28cb74a570dcf962dd9f1fe13cb815122a507f24bed6ad0e4043d3ab30e2b22a"}
Sep 05 02:56:21.346401 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.346324    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"
Sep 05 02:56:21.353507 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.353487    2557 scope.go:117] "RemoveContainer" containerID="b77c7d4b9e38f782d7c746d4712553cfda6b7811874abfd2658c48142ef6c399"
Sep 05 02:56:21.353753 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:56:21.353734    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77c7d4b9e38f782d7c746d4712553cfda6b7811874abfd2658c48142ef6c399\": container with ID starting with b77c7d4b9e38f782d7c746d4712553cfda6b7811874abfd2658c48142ef6c399 not found: ID does not exist" containerID="b77c7d4b9e38f782d7c746d4712553cfda6b7811874abfd2658c48142ef6c399"
Sep 05 02:56:21.353835 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.353763    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77c7d4b9e38f782d7c746d4712553cfda6b7811874abfd2658c48142ef6c399"} err="failed to get container status \"b77c7d4b9e38f782d7c746d4712553cfda6b7811874abfd2658c48142ef6c399\": rpc error: code = NotFound desc = could not find container \"b77c7d4b9e38f782d7c746d4712553cfda6b7811874abfd2658c48142ef6c399\": container with ID starting with b77c7d4b9e38f782d7c746d4712553cfda6b7811874abfd2658c48142ef6c399 not found: ID does not exist"
Sep 05 02:56:21.353835 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.353784    2557 scope.go:117] "RemoveContainer" containerID="5badd9ecf12818319bbe022d84b0727e7fbfaebdb70029f11a90ddd8bfab1787"
Sep 05 02:56:21.361245 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.361230    2557 scope.go:117] "RemoveContainer" containerID="5badd9ecf12818319bbe022d84b0727e7fbfaebdb70029f11a90ddd8bfab1787"
Sep 05 02:56:21.361485 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:56:21.361469    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5badd9ecf12818319bbe022d84b0727e7fbfaebdb70029f11a90ddd8bfab1787\": container with ID starting with 5badd9ecf12818319bbe022d84b0727e7fbfaebdb70029f11a90ddd8bfab1787 not found: ID does not exist" containerID="5badd9ecf12818319bbe022d84b0727e7fbfaebdb70029f11a90ddd8bfab1787"
Sep 05 02:56:21.361529 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.361491    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5badd9ecf12818319bbe022d84b0727e7fbfaebdb70029f11a90ddd8bfab1787"} err="failed to get container status \"5badd9ecf12818319bbe022d84b0727e7fbfaebdb70029f11a90ddd8bfab1787\": rpc error: code = NotFound desc = could not find container \"5badd9ecf12818319bbe022d84b0727e7fbfaebdb70029f11a90ddd8bfab1787\": container with ID starting with 5badd9ecf12818319bbe022d84b0727e7fbfaebdb70029f11a90ddd8bfab1787 not found: ID does not exist"
Sep 05 02:56:21.366910 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.366883    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"]
Sep 05 02:56:21.369558 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.369528    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2gqzw"]
Sep 05 02:56:21.379953 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.379931    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"]
Sep 05 02:56:21.383574 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.383521    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-56w75"]
Sep 05 02:56:21.414050 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.414027    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ce2c66-820f-4486-862f-d0d3f24aabd8-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:21.414050 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.414048    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ce2c66-820f-4486-862f-d0d3f24aabd8-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:21.414165 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:21.414063    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xstwt\" (UniqueName: \"kubernetes.io/projected/93ce2c66-820f-4486-862f-d0d3f24aabd8-kube-api-access-xstwt\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:56:22.990024 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:22.989991    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b8f113-83f3-45e2-a6ac-fee5026d4ea2" path="/var/lib/kubelet/pods/37b8f113-83f3-45e2-a6ac-fee5026d4ea2/volumes"
Sep 05 02:56:22.990475 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:22.990373    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93ce2c66-820f-4486-862f-d0d3f24aabd8" path="/var/lib/kubelet/pods/93ce2c66-820f-4486-862f-d0d3f24aabd8/volumes"
Sep 05 02:56:26.529084 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:26.529044    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:26.543234 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:26.543209    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:27.389830 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:27.389797    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:56:42.770787 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:42.770752    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:42.871419 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:42.871378    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:46.277328 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:46.277295    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:48.737246 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:48.737204    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:51.790006 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.789965    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:51.797602 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.797569    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"]
Sep 05 02:56:51.797939 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.797927    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ce2c66-820f-4486-862f-d0d3f24aabd8" containerName="main"
Sep 05 02:56:51.797983 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.797941    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ce2c66-820f-4486-862f-d0d3f24aabd8" containerName="main"
Sep 05 02:56:51.797983 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.797964    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b8f113-83f3-45e2-a6ac-fee5026d4ea2" containerName="main"
Sep 05 02:56:51.797983 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.797970    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b8f113-83f3-45e2-a6ac-fee5026d4ea2" containerName="main"
Sep 05 02:56:51.797983 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.797984    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc" containerName="main"
Sep 05 02:56:51.798112 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.797990    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc" containerName="main"
Sep 05 02:56:51.798112 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.798043    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b8f113-83f3-45e2-a6ac-fee5026d4ea2" containerName="main"
Sep 05 02:56:51.798112 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.798056    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a5bf0b5-954a-4131-b4b7-c5606aa4c8bc" containerName="main"
Sep 05 02:56:51.798112 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.798062    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="93ce2c66-820f-4486-862f-d0d3f24aabd8" containerName="main"
Sep 05 02:56:51.801276 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.801260    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.806325 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.806297    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 05 02:56:51.806482 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.806308    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 05 02:56:51.806557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.806484    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 05 02:56:51.806557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.806520    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 05 02:56:51.807093 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.807075    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-86nxw\""
Sep 05 02:56:51.807244 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.807220    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:56:51.834371 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.834334    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"]
Sep 05 02:56:51.872899 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.872863    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.873102 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.872916    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-tmp-dir\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.873102 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.872942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8307f7-ee37-4051-b676-f28d19b7b7c9-tls-certs\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.873102 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.872963    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b8307f7-ee37-4051-b676-f28d19b7b7c9-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.873102 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.873008    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-dshm\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.873102 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.873059    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-home\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.873102 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.873080    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-model-cache\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.873371 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.873107    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgkzl\" (UniqueName: \"kubernetes.io/projected/9b8307f7-ee37-4051-b676-f28d19b7b7c9-kube-api-access-rgkzl\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.973721 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.973678    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.973924 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.973731    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-tmp-dir\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.973924 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.973767    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8307f7-ee37-4051-b676-f28d19b7b7c9-tls-certs\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.973924 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.973789    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b8307f7-ee37-4051-b676-f28d19b7b7c9-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.973924 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.973845    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-dshm\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.973924 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.973897    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-home\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.974210 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.973927    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-model-cache\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.974210 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.973974    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgkzl\" (UniqueName: \"kubernetes.io/projected/9b8307f7-ee37-4051-b676-f28d19b7b7c9-kube-api-access-rgkzl\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.974210 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.974183    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.974377 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.974230    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-tmp-dir\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.974377 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.974266    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-home\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.974377 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.974317    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-model-cache\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.974769 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.974749    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b8307f7-ee37-4051-b676-f28d19b7b7c9-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.976231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.976210    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-dshm\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.976489 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.976469    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8307f7-ee37-4051-b676-f28d19b7b7c9-tls-certs\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:51.982181 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:51.982160    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgkzl\" (UniqueName: \"kubernetes.io/projected/9b8307f7-ee37-4051-b676-f28d19b7b7c9-kube-api-access-rgkzl\") pod \"tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:52.111744 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.111680    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:56:52.114912 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.114867    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"]
Sep 05 02:56:52.119137 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.119112    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:52.129056 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.128755    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"]
Sep 05 02:56:52.164748 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.164674    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:52.224943 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.224908    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:52.277026 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.276985    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44cebf7b-8713-4774-9a14-908e847fe6c1-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:52.277260 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.277136    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cebf7b-8713-4774-9a14-908e847fe6c1-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:52.277260 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.277194    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44cebf7b-8713-4774-9a14-908e847fe6c1-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:52.277407 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.277267    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44cebf7b-8713-4774-9a14-908e847fe6c1-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:52.378283 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.378182    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cebf7b-8713-4774-9a14-908e847fe6c1-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:52.378283 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.378240    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44cebf7b-8713-4774-9a14-908e847fe6c1-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:52.378560 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.378289    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44cebf7b-8713-4774-9a14-908e847fe6c1-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:52.378560 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.378332    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44cebf7b-8713-4774-9a14-908e847fe6c1-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:52.378714 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.378686    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cebf7b-8713-4774-9a14-908e847fe6c1-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:52.378809 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.378757    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44cebf7b-8713-4774-9a14-908e847fe6c1-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:52.379100 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.379077    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44cebf7b-8713-4774-9a14-908e847fe6c1-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:52.381014 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.380988    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44cebf7b-8713-4774-9a14-908e847fe6c1-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:52.407949 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.407918    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"]
Sep 05 02:56:52.409394 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:56:52.409369    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8307f7_ee37_4051_b676_f28d19b7b7c9.slice/crio-9b3dcded094dcc2e122cf7adb38251b2506be2bf36883dcfe589c1849b29cbb0 WatchSource:0}: Error finding container 9b3dcded094dcc2e122cf7adb38251b2506be2bf36883dcfe589c1849b29cbb0: Status 404 returned error can't find the container with id 9b3dcded094dcc2e122cf7adb38251b2506be2bf36883dcfe589c1849b29cbb0
Sep 05 02:56:52.454356 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.454318    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:52.531148 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.531106    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6" event={"ID":"9b8307f7-ee37-4051-b676-f28d19b7b7c9","Type":"ContainerStarted","Data":"0b65c38769bb1f0e0c9e1727b482fd736990a38bc6d00454f4eb8fe9d2cbd36c"}
Sep 05 02:56:52.531321 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.531218    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6" event={"ID":"9b8307f7-ee37-4051-b676-f28d19b7b7c9","Type":"ContainerStarted","Data":"9b3dcded094dcc2e122cf7adb38251b2506be2bf36883dcfe589c1849b29cbb0"}
Sep 05 02:56:52.608998 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.608962    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:52.715555 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.715467    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:52.731165 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:52.731130    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"]
Sep 05 02:56:52.732875 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:56:52.732845    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44cebf7b_8713_4774_9a14_908e847fe6c1.slice/crio-3d3c9e945ed01c9d3d112ab48d3154066ae692cb5253285d56a4a4d606d3733f WatchSource:0}: Error finding container 3d3c9e945ed01c9d3d112ab48d3154066ae692cb5253285d56a4a4d606d3733f: Status 404 returned error can't find the container with id 3d3c9e945ed01c9d3d112ab48d3154066ae692cb5253285d56a4a4d606d3733f
Sep 05 02:56:53.543468 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:53.538883    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s" event={"ID":"44cebf7b-8713-4774-9a14-908e847fe6c1","Type":"ContainerStarted","Data":"d9d9cf0b2c5abeab4f4613ee2a2b3133822eb2c200a52a4c46d93a7fd69a32e8"}
Sep 05 02:56:53.543468 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:53.538954    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s" event={"ID":"44cebf7b-8713-4774-9a14-908e847fe6c1","Type":"ContainerStarted","Data":"3d3c9e945ed01c9d3d112ab48d3154066ae692cb5253285d56a4a4d606d3733f"}
Sep 05 02:56:54.870539 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:54.870497    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:54.941843 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:54.941798    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:56:55.554261 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:55.554223    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b8307f7-ee37-4051-b676-f28d19b7b7c9" containerID="0b65c38769bb1f0e0c9e1727b482fd736990a38bc6d00454f4eb8fe9d2cbd36c" exitCode=0
Sep 05 02:56:55.554448 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:55.554295    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6" event={"ID":"9b8307f7-ee37-4051-b676-f28d19b7b7c9","Type":"ContainerDied","Data":"0b65c38769bb1f0e0c9e1727b482fd736990a38bc6d00454f4eb8fe9d2cbd36c"}
Sep 05 02:56:55.556194 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:55.556171    2557 generic.go:356] "Generic (PLEG): container finished" podID="44cebf7b-8713-4774-9a14-908e847fe6c1" containerID="d9d9cf0b2c5abeab4f4613ee2a2b3133822eb2c200a52a4c46d93a7fd69a32e8" exitCode=0
Sep 05 02:56:55.556303 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:55.556226    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s" event={"ID":"44cebf7b-8713-4774-9a14-908e847fe6c1","Type":"ContainerDied","Data":"d9d9cf0b2c5abeab4f4613ee2a2b3133822eb2c200a52a4c46d93a7fd69a32e8"}
Sep 05 02:56:56.563849 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:56.563806    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6" event={"ID":"9b8307f7-ee37-4051-b676-f28d19b7b7c9","Type":"ContainerStarted","Data":"2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd"}
Sep 05 02:56:56.566034 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:56.566004    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s" event={"ID":"44cebf7b-8713-4774-9a14-908e847fe6c1","Type":"ContainerStarted","Data":"00b65ab4837ce39bf6510b4743308b91eb544f8aa778bc1099fc2de802c3d118"}
Sep 05 02:56:56.566255 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:56.566233    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:56:56.595569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:56.595505    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6" podStartSLOduration=2.922906159 podStartE2EDuration="5.59548299s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.672576831s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:56:56.591002483 +0000 UTC m=+2320.146158793" watchObservedRunningTime="2026-09-05 02:56:56.59548299 +0000 UTC m=+2320.150639301"
Sep 05 02:56:56.637551 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:56.637481    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s" podStartSLOduration=2.004894125 podStartE2EDuration="4.637420438s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.632526313s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:56:56.636591026 +0000 UTC m=+2320.191747336" watchObservedRunningTime="2026-09-05 02:56:56.637420438 +0000 UTC m=+2320.192576748"
Sep 05 02:56:57.112638 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:56:57.112591    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:57:02.112638 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:02.112595    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:57:02.125730 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:02.125698    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:57:02.620310 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:02.620277    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:57:20.024832 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:20.024728    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:57:21.277730 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:21.277685    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:57:27.586557 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:27.586521    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:57:36.838581 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:36.838539    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:57:39.724028 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:39.723994    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s_44cebf7b-8713-4774-9a14-908e847fe6c1/main/0.log"
Sep 05 02:57:54.033794 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.033722    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"]
Sep 05 02:57:54.034290 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.034029    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s" podUID="44cebf7b-8713-4774-9a14-908e847fe6c1" containerName="main" containerID="cri-o://00b65ab4837ce39bf6510b4743308b91eb544f8aa778bc1099fc2de802c3d118" gracePeriod=30
Sep 05 02:57:54.035544 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.035514    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:57:54.037601 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.037577    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"]
Sep 05 02:57:54.048454 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:57:54.048410    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 02:57:54.048454 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:57:54.048454 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:57:54.048454 ip-10-0-143-128 kubenswrapper[2557]:  > containerID="2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd" cmd=["/bin/sleep","15"]
Sep 05 02:57:54.048705 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.048473    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6" message="" err=<
Sep 05 02:57:54.048705 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:57:54.048705 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:57:54.048705 ip-10-0-143-128 kubenswrapper[2557]:  >
Sep 05 02:57:54.048705 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:57:54.048487    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 02:57:54.048705 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 02:57:54.048705 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 02:57:54.048705 ip-10-0-143-128 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6" podUID="9b8307f7-ee37-4051-b676-f28d19b7b7c9" containerName="main" containerID="cri-o://2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd"
Sep 05 02:57:54.048705 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.048521    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6" podUID="9b8307f7-ee37-4051-b676-f28d19b7b7c9" containerName="main" containerID="cri-o://2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd" gracePeriod=60
Sep 05 02:57:54.105107 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.105074    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:57:54.605027 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.604982    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6" podUID="9b8307f7-ee37-4051-b676-f28d19b7b7c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 05 02:57:54.691147 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.691121    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:57:54.811791 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.811760    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:57:54.848011 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.847975    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44cebf7b-8713-4774-9a14-908e847fe6c1-cabundle-cert\") pod \"44cebf7b-8713-4774-9a14-908e847fe6c1\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") "
Sep 05 02:57:54.848337 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.848309    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cebf7b-8713-4774-9a14-908e847fe6c1-tmp-dir\") pod \"44cebf7b-8713-4774-9a14-908e847fe6c1\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") "
Sep 05 02:57:54.848546 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.848514    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44cebf7b-8713-4774-9a14-908e847fe6c1-kserve-provision-location\") pod \"44cebf7b-8713-4774-9a14-908e847fe6c1\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") "
Sep 05 02:57:54.848672 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.848553    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44cebf7b-8713-4774-9a14-908e847fe6c1-tls-certs\") pod \"44cebf7b-8713-4774-9a14-908e847fe6c1\" (UID: \"44cebf7b-8713-4774-9a14-908e847fe6c1\") "
Sep 05 02:57:54.848672 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.848399    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44cebf7b-8713-4774-9a14-908e847fe6c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "44cebf7b-8713-4774-9a14-908e847fe6c1" (UID: "44cebf7b-8713-4774-9a14-908e847fe6c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:57:54.848884 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.848863    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44cebf7b-8713-4774-9a14-908e847fe6c1-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:57:54.849495 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.849468    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cebf7b-8713-4774-9a14-908e847fe6c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44cebf7b-8713-4774-9a14-908e847fe6c1" (UID: "44cebf7b-8713-4774-9a14-908e847fe6c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:57:54.853117 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.852693    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44cebf7b-8713-4774-9a14-908e847fe6c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44cebf7b-8713-4774-9a14-908e847fe6c1" (UID: "44cebf7b-8713-4774-9a14-908e847fe6c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:57:54.876331 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.876296    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cebf7b-8713-4774-9a14-908e847fe6c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "44cebf7b-8713-4774-9a14-908e847fe6c1" (UID: "44cebf7b-8713-4774-9a14-908e847fe6c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:57:54.898479 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.898429    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b8307f7-ee37-4051-b676-f28d19b7b7c9" containerID="2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd" exitCode=0
Sep 05 02:57:54.898628 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.898501    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"
Sep 05 02:57:54.898628 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.898539    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6" event={"ID":"9b8307f7-ee37-4051-b676-f28d19b7b7c9","Type":"ContainerDied","Data":"2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd"}
Sep 05 02:57:54.898628 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.898609    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6" event={"ID":"9b8307f7-ee37-4051-b676-f28d19b7b7c9","Type":"ContainerDied","Data":"9b3dcded094dcc2e122cf7adb38251b2506be2bf36883dcfe589c1849b29cbb0"}
Sep 05 02:57:54.898808 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.898640    2557 scope.go:117] "RemoveContainer" containerID="2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd"
Sep 05 02:57:54.900258 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.900231    2557 generic.go:356] "Generic (PLEG): container finished" podID="44cebf7b-8713-4774-9a14-908e847fe6c1" containerID="00b65ab4837ce39bf6510b4743308b91eb544f8aa778bc1099fc2de802c3d118" exitCode=0
Sep 05 02:57:54.900364 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.900276    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"
Sep 05 02:57:54.900364 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.900272    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s" event={"ID":"44cebf7b-8713-4774-9a14-908e847fe6c1","Type":"ContainerDied","Data":"00b65ab4837ce39bf6510b4743308b91eb544f8aa778bc1099fc2de802c3d118"}
Sep 05 02:57:54.900523 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.900395    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s" event={"ID":"44cebf7b-8713-4774-9a14-908e847fe6c1","Type":"ContainerDied","Data":"3d3c9e945ed01c9d3d112ab48d3154066ae692cb5253285d56a4a4d606d3733f"}
Sep 05 02:57:54.909933 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.909916    2557 scope.go:117] "RemoveContainer" containerID="0b65c38769bb1f0e0c9e1727b482fd736990a38bc6d00454f4eb8fe9d2cbd36c"
Sep 05 02:57:54.918605 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.918584    2557 scope.go:117] "RemoveContainer" containerID="2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd"
Sep 05 02:57:54.918893 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:57:54.918875    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd\": container with ID starting with 2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd not found: ID does not exist" containerID="2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd"
Sep 05 02:57:54.918945 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.918900    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd"} err="failed to get container status \"2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd\": rpc error: code = NotFound desc = could not find container \"2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd\": container with ID starting with 2ba8d755260653bcc4c5b2681830f17d6d956d08c39a72af6351a4c69272fabd not found: ID does not exist"
Sep 05 02:57:54.918945 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.918917    2557 scope.go:117] "RemoveContainer" containerID="0b65c38769bb1f0e0c9e1727b482fd736990a38bc6d00454f4eb8fe9d2cbd36c"
Sep 05 02:57:54.919149 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:57:54.919130    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b65c38769bb1f0e0c9e1727b482fd736990a38bc6d00454f4eb8fe9d2cbd36c\": container with ID starting with 0b65c38769bb1f0e0c9e1727b482fd736990a38bc6d00454f4eb8fe9d2cbd36c not found: ID does not exist" containerID="0b65c38769bb1f0e0c9e1727b482fd736990a38bc6d00454f4eb8fe9d2cbd36c"
Sep 05 02:57:54.919203 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.919160    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b65c38769bb1f0e0c9e1727b482fd736990a38bc6d00454f4eb8fe9d2cbd36c"} err="failed to get container status \"0b65c38769bb1f0e0c9e1727b482fd736990a38bc6d00454f4eb8fe9d2cbd36c\": rpc error: code = NotFound desc = could not find container \"0b65c38769bb1f0e0c9e1727b482fd736990a38bc6d00454f4eb8fe9d2cbd36c\": container with ID starting with 0b65c38769bb1f0e0c9e1727b482fd736990a38bc6d00454f4eb8fe9d2cbd36c not found: ID does not exist"
Sep 05 02:57:54.919203 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.919175    2557 scope.go:117] "RemoveContainer" containerID="00b65ab4837ce39bf6510b4743308b91eb544f8aa778bc1099fc2de802c3d118"
Sep 05 02:57:54.924114 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.924092    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"]
Sep 05 02:57:54.928573 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.928550    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7c877dfcf4-mdm8s"]
Sep 05 02:57:54.928809 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.928795    2557 scope.go:117] "RemoveContainer" containerID="d9d9cf0b2c5abeab4f4613ee2a2b3133822eb2c200a52a4c46d93a7fd69a32e8"
Sep 05 02:57:54.936718 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.936700    2557 scope.go:117] "RemoveContainer" containerID="00b65ab4837ce39bf6510b4743308b91eb544f8aa778bc1099fc2de802c3d118"
Sep 05 02:57:54.936995 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:57:54.936978    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b65ab4837ce39bf6510b4743308b91eb544f8aa778bc1099fc2de802c3d118\": container with ID starting with 00b65ab4837ce39bf6510b4743308b91eb544f8aa778bc1099fc2de802c3d118 not found: ID does not exist" containerID="00b65ab4837ce39bf6510b4743308b91eb544f8aa778bc1099fc2de802c3d118"
Sep 05 02:57:54.937059 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.936999    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b65ab4837ce39bf6510b4743308b91eb544f8aa778bc1099fc2de802c3d118"} err="failed to get container status \"00b65ab4837ce39bf6510b4743308b91eb544f8aa778bc1099fc2de802c3d118\": rpc error: code = NotFound desc = could not find container \"00b65ab4837ce39bf6510b4743308b91eb544f8aa778bc1099fc2de802c3d118\": container with ID starting with 00b65ab4837ce39bf6510b4743308b91eb544f8aa778bc1099fc2de802c3d118 not found: ID does not exist"
Sep 05 02:57:54.937059 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.937014    2557 scope.go:117] "RemoveContainer" containerID="d9d9cf0b2c5abeab4f4613ee2a2b3133822eb2c200a52a4c46d93a7fd69a32e8"
Sep 05 02:57:54.937267 ip-10-0-143-128 kubenswrapper[2557]: E0905 02:57:54.937248    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d9cf0b2c5abeab4f4613ee2a2b3133822eb2c200a52a4c46d93a7fd69a32e8\": container with ID starting with d9d9cf0b2c5abeab4f4613ee2a2b3133822eb2c200a52a4c46d93a7fd69a32e8 not found: ID does not exist" containerID="d9d9cf0b2c5abeab4f4613ee2a2b3133822eb2c200a52a4c46d93a7fd69a32e8"
Sep 05 02:57:54.937310 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.937274    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d9cf0b2c5abeab4f4613ee2a2b3133822eb2c200a52a4c46d93a7fd69a32e8"} err="failed to get container status \"d9d9cf0b2c5abeab4f4613ee2a2b3133822eb2c200a52a4c46d93a7fd69a32e8\": rpc error: code = NotFound desc = could not find container \"d9d9cf0b2c5abeab4f4613ee2a2b3133822eb2c200a52a4c46d93a7fd69a32e8\": container with ID starting with d9d9cf0b2c5abeab4f4613ee2a2b3133822eb2c200a52a4c46d93a7fd69a32e8 not found: ID does not exist"
Sep 05 02:57:54.949583 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.949556    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-tmp-dir\") pod \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") "
Sep 05 02:57:54.949682 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.949621    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8307f7-ee37-4051-b676-f28d19b7b7c9-tls-certs\") pod \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") "
Sep 05 02:57:54.949682 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.949670    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b8307f7-ee37-4051-b676-f28d19b7b7c9-cabundle-cert\") pod \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") "
Sep 05 02:57:54.949788 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.949691    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-dshm\") pod \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") "
Sep 05 02:57:54.949788 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.949708    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-model-cache\") pod \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") "
Sep 05 02:57:54.949788 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.949741    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-kserve-provision-location\") pod \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") "
Sep 05 02:57:54.949788 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.949779    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-home\") pod \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") "
Sep 05 02:57:54.949997 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.949801    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgkzl\" (UniqueName: \"kubernetes.io/projected/9b8307f7-ee37-4051-b676-f28d19b7b7c9-kube-api-access-rgkzl\") pod \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\" (UID: \"9b8307f7-ee37-4051-b676-f28d19b7b7c9\") "
Sep 05 02:57:54.949997 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.949827    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b8307f7-ee37-4051-b676-f28d19b7b7c9" (UID: "9b8307f7-ee37-4051-b676-f28d19b7b7c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:57:54.950103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.950029    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b8307f7-ee37-4051-b676-f28d19b7b7c9" (UID: "9b8307f7-ee37-4051-b676-f28d19b7b7c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:57:54.950103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.950059    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:57:54.950103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.950067    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-home" (OuterVolumeSpecName: "home") pod "9b8307f7-ee37-4051-b676-f28d19b7b7c9" (UID: "9b8307f7-ee37-4051-b676-f28d19b7b7c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:57:54.950103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.950079    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cebf7b-8713-4774-9a14-908e847fe6c1-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:57:54.950103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.950093    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44cebf7b-8713-4774-9a14-908e847fe6c1-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:57:54.950336 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.950099    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8307f7-ee37-4051-b676-f28d19b7b7c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b8307f7-ee37-4051-b676-f28d19b7b7c9" (UID: "9b8307f7-ee37-4051-b676-f28d19b7b7c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:57:54.950336 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.950108    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44cebf7b-8713-4774-9a14-908e847fe6c1-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:57:54.951789 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.951762    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b8307f7-ee37-4051-b676-f28d19b7b7c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b8307f7-ee37-4051-b676-f28d19b7b7c9" (UID: "9b8307f7-ee37-4051-b676-f28d19b7b7c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:57:54.952081 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.952059    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8307f7-ee37-4051-b676-f28d19b7b7c9-kube-api-access-rgkzl" (OuterVolumeSpecName: "kube-api-access-rgkzl") pod "9b8307f7-ee37-4051-b676-f28d19b7b7c9" (UID: "9b8307f7-ee37-4051-b676-f28d19b7b7c9"). InnerVolumeSpecName "kube-api-access-rgkzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:57:54.952264 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.952245    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-dshm" (OuterVolumeSpecName: "dshm") pod "9b8307f7-ee37-4051-b676-f28d19b7b7c9" (UID: "9b8307f7-ee37-4051-b676-f28d19b7b7c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:57:54.973022 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.972961    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b8307f7-ee37-4051-b676-f28d19b7b7c9" (UID: "9b8307f7-ee37-4051-b676-f28d19b7b7c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:57:54.992014 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:54.991988    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44cebf7b-8713-4774-9a14-908e847fe6c1" path="/var/lib/kubelet/pods/44cebf7b-8713-4774-9a14-908e847fe6c1/volumes"
Sep 05 02:57:55.050782 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:55.050744    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8307f7-ee37-4051-b676-f28d19b7b7c9-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:57:55.050782 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:55.050778    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b8307f7-ee37-4051-b676-f28d19b7b7c9-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:57:55.050782 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:55.050788    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:57:55.051370 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:55.050797    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:57:55.051370 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:55.050806    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:57:55.051370 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:55.050815    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b8307f7-ee37-4051-b676-f28d19b7b7c9-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:57:55.051370 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:55.050826    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgkzl\" (UniqueName: \"kubernetes.io/projected/9b8307f7-ee37-4051-b676-f28d19b7b7c9-kube-api-access-rgkzl\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:57:55.214007 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:55.213972    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"]
Sep 05 02:57:55.220394 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:55.220365    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-bfcd88bdd-bbjf6"]
Sep 05 02:57:56.990868 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:57:56.990836    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b8307f7-ee37-4051-b676-f28d19b7b7c9" path="/var/lib/kubelet/pods/9b8307f7-ee37-4051-b676-f28d19b7b7c9/volumes"
Sep 05 02:58:02.701929 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:02.701894    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:58:06.696081 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:06.696036    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:58:17.122003 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:17.121974    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:58:17.133034 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:17.133007    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 02:58:20.242381 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:20.242344    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:58:20.362432 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:20.362393    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:58:20.444933 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:20.444898    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:58:20.900408 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:20.900368    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"]
Sep 05 02:58:20.927878 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:20.927839    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:58:25.317839 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:25.317802    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:58:35.921089 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:35.921045    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl" podUID="0fb8e043-2935-4e19-b01a-ccaf34c43046" containerName="main" containerID="cri-o://5e62b6a8e52e7b70707ad0cf4366b9d867d3b7bcad23ab0fd99535deb594e8e7" gracePeriod=45
Sep 05 02:58:36.115823 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.115771    2557 generic.go:356] "Generic (PLEG): container finished" podID="0fb8e043-2935-4e19-b01a-ccaf34c43046" containerID="5e62b6a8e52e7b70707ad0cf4366b9d867d3b7bcad23ab0fd99535deb594e8e7" exitCode=0
Sep 05 02:58:36.115992 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.115844    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl" event={"ID":"0fb8e043-2935-4e19-b01a-ccaf34c43046","Type":"ContainerDied","Data":"5e62b6a8e52e7b70707ad0cf4366b9d867d3b7bcad23ab0fd99535deb594e8e7"}
Sep 05 02:58:36.171801 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.171748    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:58:36.320669 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.320639    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x58l5\" (UniqueName: \"kubernetes.io/projected/0fb8e043-2935-4e19-b01a-ccaf34c43046-kube-api-access-x58l5\") pod \"0fb8e043-2935-4e19-b01a-ccaf34c43046\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") "
Sep 05 02:58:36.320860 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.320691    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-dshm\") pod \"0fb8e043-2935-4e19-b01a-ccaf34c43046\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") "
Sep 05 02:58:36.320860 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.320728    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb8e043-2935-4e19-b01a-ccaf34c43046-tls-certs\") pod \"0fb8e043-2935-4e19-b01a-ccaf34c43046\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") "
Sep 05 02:58:36.320860 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.320781    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-model-cache\") pod \"0fb8e043-2935-4e19-b01a-ccaf34c43046\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") "
Sep 05 02:58:36.320860 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.320799    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-home\") pod \"0fb8e043-2935-4e19-b01a-ccaf34c43046\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") "
Sep 05 02:58:36.320860 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.320815    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-tmp-dir\") pod \"0fb8e043-2935-4e19-b01a-ccaf34c43046\" (UID: \"0fb8e043-2935-4e19-b01a-ccaf34c43046\") "
Sep 05 02:58:36.321125 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.321079    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-home" (OuterVolumeSpecName: "home") pod "0fb8e043-2935-4e19-b01a-ccaf34c43046" (UID: "0fb8e043-2935-4e19-b01a-ccaf34c43046"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:58:36.321125 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.321084    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-model-cache" (OuterVolumeSpecName: "model-cache") pod "0fb8e043-2935-4e19-b01a-ccaf34c43046" (UID: "0fb8e043-2935-4e19-b01a-ccaf34c43046"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:58:36.321242 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.321131    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0fb8e043-2935-4e19-b01a-ccaf34c43046" (UID: "0fb8e043-2935-4e19-b01a-ccaf34c43046"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:58:36.323009 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.322980    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-dshm" (OuterVolumeSpecName: "dshm") pod "0fb8e043-2935-4e19-b01a-ccaf34c43046" (UID: "0fb8e043-2935-4e19-b01a-ccaf34c43046"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:58:36.323120 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.323038    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb8e043-2935-4e19-b01a-ccaf34c43046-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fb8e043-2935-4e19-b01a-ccaf34c43046" (UID: "0fb8e043-2935-4e19-b01a-ccaf34c43046"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:58:36.323120 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.323062    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb8e043-2935-4e19-b01a-ccaf34c43046-kube-api-access-x58l5" (OuterVolumeSpecName: "kube-api-access-x58l5") pod "0fb8e043-2935-4e19-b01a-ccaf34c43046" (UID: "0fb8e043-2935-4e19-b01a-ccaf34c43046"). InnerVolumeSpecName "kube-api-access-x58l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:58:36.421873 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.421792    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb8e043-2935-4e19-b01a-ccaf34c43046-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:58:36.421873 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.421821    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:58:36.421873 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.421831    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:58:36.421873 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.421839    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:58:36.421873 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.421848    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x58l5\" (UniqueName: \"kubernetes.io/projected/0fb8e043-2935-4e19-b01a-ccaf34c43046-kube-api-access-x58l5\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:58:36.421873 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:36.421857    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fb8e043-2935-4e19-b01a-ccaf34c43046-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 02:58:37.122360 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:37.122330    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"
Sep 05 02:58:37.122742 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:37.122328    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl" event={"ID":"0fb8e043-2935-4e19-b01a-ccaf34c43046","Type":"ContainerDied","Data":"ceab8c88276df66065cdc4e89990d049ca989f9b9715d174dccc53e07409a6f7"}
Sep 05 02:58:37.122742 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:37.122491    2557 scope.go:117] "RemoveContainer" containerID="5e62b6a8e52e7b70707ad0cf4366b9d867d3b7bcad23ab0fd99535deb594e8e7"
Sep 05 02:58:37.139219 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:37.139192    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"]
Sep 05 02:58:37.141232 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:37.141211    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-csqxl"]
Sep 05 02:58:38.991329 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:38.991250    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fb8e043-2935-4e19-b01a-ccaf34c43046" path="/var/lib/kubelet/pods/0fb8e043-2935-4e19-b01a-ccaf34c43046/volumes"
Sep 05 02:58:40.555281 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:40.555247    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:58:40.647174 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:40.647132    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:58:46.698332 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:46.698294    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:58:52.265820 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:52.265781    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:58:52.367549 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:52.367510    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:58:58.145652 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:58:58.145619    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:59:02.322468 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:02.322421    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:59:07.925415 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:07.925363    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:59:07.994683 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:07.994651    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:59:12.601569 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.601502    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"]
Sep 05 02:59:12.602019 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.601982    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44cebf7b-8713-4774-9a14-908e847fe6c1" containerName="storage-initializer"
Sep 05 02:59:12.602019 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.602007    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="44cebf7b-8713-4774-9a14-908e847fe6c1" containerName="storage-initializer"
Sep 05 02:59:12.602103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.602021    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44cebf7b-8713-4774-9a14-908e847fe6c1" containerName="main"
Sep 05 02:59:12.602103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.602029    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="44cebf7b-8713-4774-9a14-908e847fe6c1" containerName="main"
Sep 05 02:59:12.602103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.602041    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b8307f7-ee37-4051-b676-f28d19b7b7c9" containerName="main"
Sep 05 02:59:12.602103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.602046    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b8307f7-ee37-4051-b676-f28d19b7b7c9" containerName="main"
Sep 05 02:59:12.602103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.602053    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fb8e043-2935-4e19-b01a-ccaf34c43046" containerName="main"
Sep 05 02:59:12.602103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.602059    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fb8e043-2935-4e19-b01a-ccaf34c43046" containerName="main"
Sep 05 02:59:12.602103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.602075    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b8307f7-ee37-4051-b676-f28d19b7b7c9" containerName="storage-initializer"
Sep 05 02:59:12.602103 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.602082    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b8307f7-ee37-4051-b676-f28d19b7b7c9" containerName="storage-initializer"
Sep 05 02:59:12.602356 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.602133    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b8307f7-ee37-4051-b676-f28d19b7b7c9" containerName="main"
Sep 05 02:59:12.602356 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.602144    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="44cebf7b-8713-4774-9a14-908e847fe6c1" containerName="main"
Sep 05 02:59:12.602356 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.602155    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fb8e043-2935-4e19-b01a-ccaf34c43046" containerName="main"
Sep 05 02:59:12.605258 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.605242    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.608412 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.608381    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 05 02:59:12.608412 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.608396    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-6f8ws\""
Sep 05 02:59:12.608412 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.608382    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 05 02:59:12.608649 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.608396    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 05 02:59:12.614380 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.614355    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"]
Sep 05 02:59:12.699539 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.699497    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:59:12.724364 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.724335    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.724524 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.724373    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr4ct\" (UniqueName: \"kubernetes.io/projected/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-kube-api-access-hr4ct\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.724570 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.724517    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.724570 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.724549    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.724640 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.724571    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-home\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.724677 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.724643    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.825395 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.825356    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.825603 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.825405    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.825603 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.825456    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-home\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.825603 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.825508    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.825603 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.825541    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.825603 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.825574    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr4ct\" (UniqueName: \"kubernetes.io/projected/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-kube-api-access-hr4ct\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.825875 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.825819    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.825925 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.825871    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-home\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.825969 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.825931    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.827722 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.827689    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.828032 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.828012    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.833492 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.833466    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr4ct\" (UniqueName: \"kubernetes.io/projected/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-kube-api-access-hr4ct\") pod \"delw-v2-kserve-5b84cc7ddf-ktb74\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:12.916562 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:12.916471    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:13.191465 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:13.191417    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"]
Sep 05 02:59:13.193060 ip-10-0-143-128 kubenswrapper[2557]: W0905 02:59:13.193034    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49fd9dfd_f4d1_4c02_a2a4_11f9d94a9380.slice/crio-e5a472a7be0939837dd3af3678ac064d8aaae9892eafd9c630e39d20298c7369 WatchSource:0}: Error finding container e5a472a7be0939837dd3af3678ac064d8aaae9892eafd9c630e39d20298c7369: Status 404 returned error can't find the container with id e5a472a7be0939837dd3af3678ac064d8aaae9892eafd9c630e39d20298c7369
Sep 05 02:59:13.317641 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:13.317591    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74" event={"ID":"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380","Type":"ContainerStarted","Data":"d79742720a9ba9ce3dc2adcaf218b31e21d99f287637ef23cf7f86493e0f34a1"}
Sep 05 02:59:13.317819 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:13.317665    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74" event={"ID":"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380","Type":"ContainerStarted","Data":"e5a472a7be0939837dd3af3678ac064d8aaae9892eafd9c630e39d20298c7369"}
Sep 05 02:59:13.335946 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:13.335885    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74" podStartSLOduration=1.335866299 podStartE2EDuration="1.335866299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:59:13.334566557 +0000 UTC m=+2456.889722900" watchObservedRunningTime="2026-09-05 02:59:13.335866299 +0000 UTC m=+2456.891022609"
Sep 05 02:59:13.917050 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:13.917002    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:21.190023 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:21.189986    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:59:21.369374 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:21.369343    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 02:59:22.917231 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:22.917191    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:22.930000 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:22.929973    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:23.384327 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:23.384291    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 02:59:35.866896 ip-10-0-143-128 kubenswrapper[2557]: I0905 02:59:35.866853    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:00:05.378246 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:05.378209    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:00:06.994094 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:06.994058    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"]
Sep 05 03:00:07.014617 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:07.014586    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:00:08.129213 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:08.129173    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:00:08.221040 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:08.221006    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:00:22.014479 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.014409    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74" podUID="49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380" containerName="main" containerID="cri-o://d79742720a9ba9ce3dc2adcaf218b31e21d99f287637ef23cf7f86493e0f34a1" gracePeriod=45
Sep 05 03:00:22.255266 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.255234    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 03:00:22.431707 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.431675    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-tmp-dir\") pod \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") "
Sep 05 03:00:22.431918 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.431721    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr4ct\" (UniqueName: \"kubernetes.io/projected/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-kube-api-access-hr4ct\") pod \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") "
Sep 05 03:00:22.431918 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.431771    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-model-cache\") pod \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") "
Sep 05 03:00:22.431918 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.431789    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-dshm\") pod \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") "
Sep 05 03:00:22.431918 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.431806    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-tls-certs\") pod \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") "
Sep 05 03:00:22.431918 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.431823    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-home\") pod \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\" (UID: \"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380\") "
Sep 05 03:00:22.432185 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.432003    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380" (UID: "49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:00:22.432185 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.432023    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-model-cache" (OuterVolumeSpecName: "model-cache") pod "49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380" (UID: "49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:00:22.432185 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.432164    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-home" (OuterVolumeSpecName: "home") pod "49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380" (UID: "49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:00:22.434217 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.434189    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-dshm" (OuterVolumeSpecName: "dshm") pod "49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380" (UID: "49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:00:22.434217 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.434202    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380" (UID: "49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:00:22.434357 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.434206    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-kube-api-access-hr4ct" (OuterVolumeSpecName: "kube-api-access-hr4ct") pod "49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380" (UID: "49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380"). InnerVolumeSpecName "kube-api-access-hr4ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:00:22.532953 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.532916    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr4ct\" (UniqueName: \"kubernetes.io/projected/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-kube-api-access-hr4ct\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:00:22.532953 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.532951    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:00:22.532953 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.532961    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:00:22.533184 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.532969    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:00:22.533184 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.532977    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:00:22.533184 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.532985    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:00:22.550941 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.550911    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:00:22.654089 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.654039    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:00:22.690210 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.690121    2557 generic.go:356] "Generic (PLEG): container finished" podID="49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380" containerID="d79742720a9ba9ce3dc2adcaf218b31e21d99f287637ef23cf7f86493e0f34a1" exitCode=0
Sep 05 03:00:22.690210 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.690180    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"
Sep 05 03:00:22.690210 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.690195    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74" event={"ID":"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380","Type":"ContainerDied","Data":"d79742720a9ba9ce3dc2adcaf218b31e21d99f287637ef23cf7f86493e0f34a1"}
Sep 05 03:00:22.690511 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.690243    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74" event={"ID":"49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380","Type":"ContainerDied","Data":"e5a472a7be0939837dd3af3678ac064d8aaae9892eafd9c630e39d20298c7369"}
Sep 05 03:00:22.690511 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.690268    2557 scope.go:117] "RemoveContainer" containerID="d79742720a9ba9ce3dc2adcaf218b31e21d99f287637ef23cf7f86493e0f34a1"
Sep 05 03:00:22.699594 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.699574    2557 scope.go:117] "RemoveContainer" containerID="d79742720a9ba9ce3dc2adcaf218b31e21d99f287637ef23cf7f86493e0f34a1"
Sep 05 03:00:22.699857 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:00:22.699836    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79742720a9ba9ce3dc2adcaf218b31e21d99f287637ef23cf7f86493e0f34a1\": container with ID starting with d79742720a9ba9ce3dc2adcaf218b31e21d99f287637ef23cf7f86493e0f34a1 not found: ID does not exist" containerID="d79742720a9ba9ce3dc2adcaf218b31e21d99f287637ef23cf7f86493e0f34a1"
Sep 05 03:00:22.699906 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.699865    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79742720a9ba9ce3dc2adcaf218b31e21d99f287637ef23cf7f86493e0f34a1"} err="failed to get container status \"d79742720a9ba9ce3dc2adcaf218b31e21d99f287637ef23cf7f86493e0f34a1\": rpc error: code = NotFound desc = could not find container \"d79742720a9ba9ce3dc2adcaf218b31e21d99f287637ef23cf7f86493e0f34a1\": container with ID starting with d79742720a9ba9ce3dc2adcaf218b31e21d99f287637ef23cf7f86493e0f34a1 not found: ID does not exist"
Sep 05 03:00:22.711862 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.711839    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"]
Sep 05 03:00:22.714600 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.714577    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:00:22.715983 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.715964    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ktb74"]
Sep 05 03:00:22.990989 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:22.990909    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380" path="/var/lib/kubelet/pods/49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380/volumes"
Sep 05 03:00:27.605606 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.605564    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:00:27.608907 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.608873    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"]
Sep 05 03:00:27.609231 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.609218    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380" containerName="main"
Sep 05 03:00:27.609288 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.609234    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380" containerName="main"
Sep 05 03:00:27.609326 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.609303    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="49fd9dfd-f4d1-4c02-a2a4-11f9d94a9380" containerName="main"
Sep 05 03:00:27.612316 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.612292    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.614406 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.614381    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 05 03:00:27.615301 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.615276    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 05 03:00:27.615467 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.615419    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Sep 05 03:00:27.615623 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.615360    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-hpctv\""
Sep 05 03:00:27.621574 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.621548    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"]
Sep 05 03:00:27.675309 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.675280    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.675309 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.675314    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-home\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.675570 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.675338    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-dshm\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.675570 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.675401    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e93c8f8-7029-4fe3-81cc-3494f883c340-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.675570 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.675419    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.675570 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.675501    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bjgt\" (UniqueName: \"kubernetes.io/projected/0e93c8f8-7029-4fe3-81cc-3494f883c340-kube-api-access-8bjgt\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.693156 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.693120    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:00:27.776510 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.776467    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-dshm\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.776704 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.776538    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e93c8f8-7029-4fe3-81cc-3494f883c340-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.776704 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.776556    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.776704 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.776610    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bjgt\" (UniqueName: \"kubernetes.io/projected/0e93c8f8-7029-4fe3-81cc-3494f883c340-kube-api-access-8bjgt\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.776704 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.776670    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.776704 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.776690    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-home\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.777017 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.776991    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.777216 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.777035    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.777368 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.777255    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-home\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.778938 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.778921    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-dshm\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.779115 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.779100    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e93c8f8-7029-4fe3-81cc-3494f883c340-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.784321 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.784301    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bjgt\" (UniqueName: \"kubernetes.io/projected/0e93c8f8-7029-4fe3-81cc-3494f883c340-kube-api-access-8bjgt\") pod \"roll-v1-kserve-69cf8bcfc-ht5f2\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:27.925604 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:27.925506    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:28.201359 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:28.201326    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"]
Sep 05 03:00:28.203214 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:00:28.203187    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e93c8f8_7029_4fe3_81cc_3494f883c340.slice/crio-f3d921aa0ba8759a9cfe41cdc01b15a1ffd0321584b658e628548a5e71259a1a WatchSource:0}: Error finding container f3d921aa0ba8759a9cfe41cdc01b15a1ffd0321584b658e628548a5e71259a1a: Status 404 returned error can't find the container with id f3d921aa0ba8759a9cfe41cdc01b15a1ffd0321584b658e628548a5e71259a1a
Sep 05 03:00:28.724114 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:28.724066    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2" event={"ID":"0e93c8f8-7029-4fe3-81cc-3494f883c340","Type":"ContainerStarted","Data":"db4e44192af4f183e8f5476b18a840e889d88f73f0a38451260033d342d1840f"}
Sep 05 03:00:28.724514 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:28.724133    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2" event={"ID":"0e93c8f8-7029-4fe3-81cc-3494f883c340","Type":"ContainerStarted","Data":"f3d921aa0ba8759a9cfe41cdc01b15a1ffd0321584b658e628548a5e71259a1a"}
Sep 05 03:00:28.741686 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:28.741630    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2" podStartSLOduration=1.741613439 podStartE2EDuration="1.741613439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:00:28.74062976 +0000 UTC m=+2532.295786070" watchObservedRunningTime="2026-09-05 03:00:28.741613439 +0000 UTC m=+2532.296769750"
Sep 05 03:00:28.926398 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:28.926363    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:33.398229 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.398190    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"]
Sep 05 03:00:33.402415 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.402391    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.404843 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.404809    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 05 03:00:33.411144 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.411098    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"]
Sep 05 03:00:33.425257 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.425226    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28aef5e4-132b-450d-84dc-022a68b721c0-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.425406 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.425264    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmf8n\" (UniqueName: \"kubernetes.io/projected/28aef5e4-132b-450d-84dc-022a68b721c0-kube-api-access-mmf8n\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.425406 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.425380    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.425541 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.425432    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-home\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.425541 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.425506    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-dshm\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.425651 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.425545    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.526176 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.526143    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-dshm\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.526339 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.526183    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.526339 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.526208    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28aef5e4-132b-450d-84dc-022a68b721c0-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.526339 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.526229    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmf8n\" (UniqueName: \"kubernetes.io/projected/28aef5e4-132b-450d-84dc-022a68b721c0-kube-api-access-mmf8n\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.526565 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.526401    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.526565 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.526468    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-home\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.526675 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.526588    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.526764 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.526743    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.526817 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.526784    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-home\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.528384 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.528366    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-dshm\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.528677 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.528660    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28aef5e4-132b-450d-84dc-022a68b721c0-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.533712 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.533687    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmf8n\" (UniqueName: \"kubernetes.io/projected/28aef5e4-132b-450d-84dc-022a68b721c0-kube-api-access-mmf8n\") pod \"roll-v2-kserve-d4dbf9d67-g6mgr\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.562239 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.562204    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:00:33.715948 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.715859    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:33.992684 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:33.992649    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"]
Sep 05 03:00:33.993939 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:00:33.993894    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28aef5e4_132b_450d_84dc_022a68b721c0.slice/crio-b00a288f769e2883e94efb719ab2c2e991576d44b6688f94435ab0b24d2a9bd9 WatchSource:0}: Error finding container b00a288f769e2883e94efb719ab2c2e991576d44b6688f94435ab0b24d2a9bd9: Status 404 returned error can't find the container with id b00a288f769e2883e94efb719ab2c2e991576d44b6688f94435ab0b24d2a9bd9
Sep 05 03:00:34.764882 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:34.764842    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr" event={"ID":"28aef5e4-132b-450d-84dc-022a68b721c0","Type":"ContainerStarted","Data":"511f87990bd7eb96aec7eb216a2517210177a1ba237aedd5c3e3d968fdecbc03"}
Sep 05 03:00:34.765286 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:34.764910    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr" event={"ID":"28aef5e4-132b-450d-84dc-022a68b721c0","Type":"ContainerStarted","Data":"b00a288f769e2883e94efb719ab2c2e991576d44b6688f94435ab0b24d2a9bd9"}
Sep 05 03:00:34.783541 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:34.783486    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr" podStartSLOduration=1.783470026 podStartE2EDuration="1.783470026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:00:34.780998148 +0000 UTC m=+2538.336154458" watchObservedRunningTime="2026-09-05 03:00:34.783470026 +0000 UTC m=+2538.338626392"
Sep 05 03:00:35.716130 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:35.716095    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:37.926014 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:37.925967    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:37.939510 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:37.939481    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:38.797118 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:38.797085    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:00:43.716985 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:43.716949    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:43.729638 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:43.729612    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:43.829656 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:43.829624    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:00:57.119580 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:57.119538    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:00:58.761068 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:58.761028    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:00:59.259849 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:00:59.259817    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:03.353102 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:03.353053    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:04.501474 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:04.501428    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:09.157728 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:09.157682    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:09.254069 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:09.254037    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:10.759095 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:10.759062    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:10.860730 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:10.860697    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:11.830086 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:11.830054    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:12.043083 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:12.043046    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:29.544689 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:29.544651    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:50.709624 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.709576    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"]
Sep 05 03:01:50.713321 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.713296    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.717076 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.717053    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 05 03:01:50.717178 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.717078    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-swzwf\""
Sep 05 03:01:50.717178 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.717078    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 05 03:01:50.717178 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.717123    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 05 03:01:50.724175 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.724147    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"]
Sep 05 03:01:50.802980 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.802944    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:50.825381 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.825349    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.825558 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.825385    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.825558 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.825427    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.825558 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.825531    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.825684 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.825569    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.825684 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.825637    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk724\" (UniqueName: \"kubernetes.io/projected/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-kube-api-access-xk724\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.926516 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.926470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.926697 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.926552    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.926697 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.926622    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk724\" (UniqueName: \"kubernetes.io/projected/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-kube-api-access-xk724\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.926697 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.926658    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.926697 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.926690    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.926925 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.926754    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.927046 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.927002    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.927166 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.927051    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.927166 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.927113    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.929219 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.929175    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.929735 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.929711    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:50.938131 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:50.938062    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk724\" (UniqueName: \"kubernetes.io/projected/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-kube-api-access-xk724\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:51.015621 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:51.015527    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:01:51.024377 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:51.024347    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:01:51.308488 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:51.308244    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"]
Sep 05 03:01:52.200348 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:52.200297    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck" event={"ID":"4a4b32e5-d7aa-49d5-80ec-749a9046eb49","Type":"ContainerStarted","Data":"ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3"}
Sep 05 03:01:52.200772 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:52.200369    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck" event={"ID":"4a4b32e5-d7aa-49d5-80ec-749a9046eb49","Type":"ContainerStarted","Data":"6dc537c6ff20bb6a4d24499205338812c52c0a68b6f7e1d4abb0fa366f2c9206"}
Sep 05 03:01:52.219935 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:52.219877    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck" podStartSLOduration=2.219862775 podStartE2EDuration="2.219862775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:01:52.218021995 +0000 UTC m=+2615.773178304" watchObservedRunningTime="2026-09-05 03:01:52.219862775 +0000 UTC m=+2615.775019084"
Sep 05 03:01:53.024503 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:01:53.024465    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:02:01.025533 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:01.025499    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:02:01.038540 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:01.038511    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:02:01.260878 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:01.260847    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:02:02.115620 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:02.115588    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:02:25.151572 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.151530    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:02:25.268996 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.268944    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:02:25.298923 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.298877    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"]
Sep 05 03:02:25.308725 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:02:25.308687    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 03:02:25.308725 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:02:25.308725 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 03:02:25.308725 ip-10-0-143-128 kubenswrapper[2557]:  > containerID="ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3" cmd=["/bin/sleep","15"]
Sep 05 03:02:25.309036 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.308747    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck" message="" err=<
Sep 05 03:02:25.309036 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:02:25.309036 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 03:02:25.309036 ip-10-0-143-128 kubenswrapper[2557]:  >
Sep 05 03:02:25.309036 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:02:25.308764    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 03:02:25.309036 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:02:25.309036 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 03:02:25.309036 ip-10-0-143-128 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck" podUID="4a4b32e5-d7aa-49d5-80ec-749a9046eb49" containerName="main" containerID="cri-o://ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3"
Sep 05 03:02:25.309036 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.308818    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck" podUID="4a4b32e5-d7aa-49d5-80ec-749a9046eb49" containerName="main" containerID="cri-o://ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3" gracePeriod=60
Sep 05 03:02:25.346323 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.346286    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:02:25.419534 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.419504    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:02:25.658694 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.658662    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:02:25.752902 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.752793    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-dshm\") pod \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") "
Sep 05 03:02:25.752902 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.752849    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk724\" (UniqueName: \"kubernetes.io/projected/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-kube-api-access-xk724\") pod \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") "
Sep 05 03:02:25.753132 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.752942    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-model-cache\") pod \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") "
Sep 05 03:02:25.753132 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.752969    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-home\") pod \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") "
Sep 05 03:02:25.753132 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.753008    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-tls-certs\") pod \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") "
Sep 05 03:02:25.753132 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.753043    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-tmp-dir\") pod \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\" (UID: \"4a4b32e5-d7aa-49d5-80ec-749a9046eb49\") "
Sep 05 03:02:25.753334 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.753176    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a4b32e5-d7aa-49d5-80ec-749a9046eb49" (UID: "4a4b32e5-d7aa-49d5-80ec-749a9046eb49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:02:25.753334 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.753272    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-home" (OuterVolumeSpecName: "home") pod "4a4b32e5-d7aa-49d5-80ec-749a9046eb49" (UID: "4a4b32e5-d7aa-49d5-80ec-749a9046eb49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:02:25.753427 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.753360    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a4b32e5-d7aa-49d5-80ec-749a9046eb49" (UID: "4a4b32e5-d7aa-49d5-80ec-749a9046eb49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:02:25.753503 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.753471    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:25.753503 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.753490    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:25.753580 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.753503    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:25.755145 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.755122    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-dshm" (OuterVolumeSpecName: "dshm") pod "4a4b32e5-d7aa-49d5-80ec-749a9046eb49" (UID: "4a4b32e5-d7aa-49d5-80ec-749a9046eb49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:02:25.755468 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.755430    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a4b32e5-d7aa-49d5-80ec-749a9046eb49" (UID: "4a4b32e5-d7aa-49d5-80ec-749a9046eb49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:02:25.755573 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.755467    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-kube-api-access-xk724" (OuterVolumeSpecName: "kube-api-access-xk724") pod "4a4b32e5-d7aa-49d5-80ec-749a9046eb49" (UID: "4a4b32e5-d7aa-49d5-80ec-749a9046eb49"). InnerVolumeSpecName "kube-api-access-xk724". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:02:25.854166 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.854128    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:25.854166 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.854162    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk724\" (UniqueName: \"kubernetes.io/projected/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-kube-api-access-xk724\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:25.854361 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:25.854179    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a4b32e5-d7aa-49d5-80ec-749a9046eb49-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:26.222531 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:26.222489    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"]
Sep 05 03:02:26.249815 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:26.249781    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:02:26.393320 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:26.393273    2557 generic.go:356] "Generic (PLEG): container finished" podID="4a4b32e5-d7aa-49d5-80ec-749a9046eb49" containerID="ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3" exitCode=0
Sep 05 03:02:26.393544 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:26.393386    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck" event={"ID":"4a4b32e5-d7aa-49d5-80ec-749a9046eb49","Type":"ContainerDied","Data":"ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3"}
Sep 05 03:02:26.393544 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:26.393400    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"
Sep 05 03:02:26.393544 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:26.393471    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck" event={"ID":"4a4b32e5-d7aa-49d5-80ec-749a9046eb49","Type":"ContainerDied","Data":"6dc537c6ff20bb6a4d24499205338812c52c0a68b6f7e1d4abb0fa366f2c9206"}
Sep 05 03:02:26.393544 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:26.393500    2557 scope.go:117] "RemoveContainer" containerID="ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3"
Sep 05 03:02:26.403477 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:26.403430    2557 scope.go:117] "RemoveContainer" containerID="ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3"
Sep 05 03:02:26.403843 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:02:26.403807    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3\": container with ID starting with ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3 not found: ID does not exist" containerID="ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3"
Sep 05 03:02:26.403951 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:26.403855    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3"} err="failed to get container status \"ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3\": rpc error: code = NotFound desc = could not find container \"ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3\": container with ID starting with ac07b7b67c9bab5baaf4f6ee0528b1d63b9f7632c17d67fbc2fd10915e8636c3 not found: ID does not exist"
Sep 05 03:02:26.422127 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:26.422096    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"]
Sep 05 03:02:26.425682 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:26.425652    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995fzdck"]
Sep 05 03:02:26.891957 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:26.891913    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"]
Sep 05 03:02:26.901486 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:26.901422    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:02:26.992633 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:26.992598    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a4b32e5-d7aa-49d5-80ec-749a9046eb49" path="/var/lib/kubelet/pods/4a4b32e5-d7aa-49d5-80ec-749a9046eb49/volumes"
Sep 05 03:02:30.209335 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:30.209298    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:02:34.426056 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.426018    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"]
Sep 05 03:02:34.426575 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.426557    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a4b32e5-d7aa-49d5-80ec-749a9046eb49" containerName="main"
Sep 05 03:02:34.426634 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.426579    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a4b32e5-d7aa-49d5-80ec-749a9046eb49" containerName="main"
Sep 05 03:02:34.426703 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.426692    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a4b32e5-d7aa-49d5-80ec-749a9046eb49" containerName="main"
Sep 05 03:02:34.431404 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.431386    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.434896 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.434691    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Sep 05 03:02:34.434896 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.434709    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 05 03:02:34.434896 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.434710    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-2z9qx\""
Sep 05 03:02:34.434896 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.434702    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 05 03:02:34.448962 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.448930    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"]
Sep 05 03:02:34.481942 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.481905    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:02:34.532996 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.532952    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-dshm\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.533173 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.533014    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a78665-9c31-4b02-bb9c-992ffcba6fe3-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.533173 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.533049    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-model-cache\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.533173 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.533081    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrnvw\" (UniqueName: \"kubernetes.io/projected/71a78665-9c31-4b02-bb9c-992ffcba6fe3-kube-api-access-lrnvw\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.533173 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.533138    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.533328 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.533180    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-home\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.634601 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.634560    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-dshm\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.634601 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.634603    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a78665-9c31-4b02-bb9c-992ffcba6fe3-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.634853 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.634622    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-model-cache\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.634853 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.634647    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrnvw\" (UniqueName: \"kubernetes.io/projected/71a78665-9c31-4b02-bb9c-992ffcba6fe3-kube-api-access-lrnvw\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.634853 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.634682    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.634853 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.634714    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-home\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.635107 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.635085    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-model-cache\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.635188 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.635141    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.635236 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.635186    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-home\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.636948 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.636928    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-dshm\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.637255 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.637237    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a78665-9c31-4b02-bb9c-992ffcba6fe3-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.647267 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.647237    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrnvw\" (UniqueName: \"kubernetes.io/projected/71a78665-9c31-4b02-bb9c-992ffcba6fe3-kube-api-access-lrnvw\") pod \"owner-v1-kserve-75579ff5bc-qk5xk\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:34.743175 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:34.743076    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:35.043026 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:35.042933    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"]
Sep 05 03:02:35.048017 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:02:35.047973    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a78665_9c31_4b02_bb9c_992ffcba6fe3.slice/crio-6c63b8e07e27016d976a8f71a18f31703009df8c87c8e56e6a14600b44b95836 WatchSource:0}: Error finding container 6c63b8e07e27016d976a8f71a18f31703009df8c87c8e56e6a14600b44b95836: Status 404 returned error can't find the container with id 6c63b8e07e27016d976a8f71a18f31703009df8c87c8e56e6a14600b44b95836
Sep 05 03:02:35.447862 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:35.447819    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk" event={"ID":"71a78665-9c31-4b02-bb9c-992ffcba6fe3","Type":"ContainerStarted","Data":"badc5ac020e22d88c79ca8e0018dfd659c5b57effa76e5843ccd1c6f9c603faf"}
Sep 05 03:02:35.448354 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:35.447880    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk" event={"ID":"71a78665-9c31-4b02-bb9c-992ffcba6fe3","Type":"ContainerStarted","Data":"6c63b8e07e27016d976a8f71a18f31703009df8c87c8e56e6a14600b44b95836"}
Sep 05 03:02:35.468485 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:35.468405    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk" podStartSLOduration=1.4683889620000001 podStartE2EDuration="1.468388962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:02:35.466584402 +0000 UTC m=+2659.021740711" watchObservedRunningTime="2026-09-05 03:02:35.468388962 +0000 UTC m=+2659.023545271"
Sep 05 03:02:35.744041 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:35.743946    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:41.248484 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.248406    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr" podUID="28aef5e4-132b-450d-84dc-022a68b721c0" containerName="main" containerID="cri-o://511f87990bd7eb96aec7eb216a2517210177a1ba237aedd5c3e3d968fdecbc03" gracePeriod=45
Sep 05 03:02:41.484631 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.484596    2557 generic.go:356] "Generic (PLEG): container finished" podID="28aef5e4-132b-450d-84dc-022a68b721c0" containerID="511f87990bd7eb96aec7eb216a2517210177a1ba237aedd5c3e3d968fdecbc03" exitCode=0
Sep 05 03:02:41.484788 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.484670    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr" event={"ID":"28aef5e4-132b-450d-84dc-022a68b721c0","Type":"ContainerDied","Data":"511f87990bd7eb96aec7eb216a2517210177a1ba237aedd5c3e3d968fdecbc03"}
Sep 05 03:02:41.507860 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.507792    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:02:41.602997 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.602961    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-home\") pod \"28aef5e4-132b-450d-84dc-022a68b721c0\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") "
Sep 05 03:02:41.603194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.603014    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmf8n\" (UniqueName: \"kubernetes.io/projected/28aef5e4-132b-450d-84dc-022a68b721c0-kube-api-access-mmf8n\") pod \"28aef5e4-132b-450d-84dc-022a68b721c0\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") "
Sep 05 03:02:41.603194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.603040    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-model-cache\") pod \"28aef5e4-132b-450d-84dc-022a68b721c0\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") "
Sep 05 03:02:41.603194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.603073    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-tmp-dir\") pod \"28aef5e4-132b-450d-84dc-022a68b721c0\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") "
Sep 05 03:02:41.603194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.603107    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28aef5e4-132b-450d-84dc-022a68b721c0-tls-certs\") pod \"28aef5e4-132b-450d-84dc-022a68b721c0\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") "
Sep 05 03:02:41.603422 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.603223    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-dshm\") pod \"28aef5e4-132b-450d-84dc-022a68b721c0\" (UID: \"28aef5e4-132b-450d-84dc-022a68b721c0\") "
Sep 05 03:02:41.603422 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.603282    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-home" (OuterVolumeSpecName: "home") pod "28aef5e4-132b-450d-84dc-022a68b721c0" (UID: "28aef5e4-132b-450d-84dc-022a68b721c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:02:41.603422 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.603386    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "28aef5e4-132b-450d-84dc-022a68b721c0" (UID: "28aef5e4-132b-450d-84dc-022a68b721c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:02:41.603422 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.603399    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28aef5e4-132b-450d-84dc-022a68b721c0" (UID: "28aef5e4-132b-450d-84dc-022a68b721c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:02:41.603645 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.603576    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:41.603645 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.603599    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:41.603645 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.603614    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:41.605455 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.605407    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-dshm" (OuterVolumeSpecName: "dshm") pod "28aef5e4-132b-450d-84dc-022a68b721c0" (UID: "28aef5e4-132b-450d-84dc-022a68b721c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:02:41.605582 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.605478    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28aef5e4-132b-450d-84dc-022a68b721c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28aef5e4-132b-450d-84dc-022a68b721c0" (UID: "28aef5e4-132b-450d-84dc-022a68b721c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:02:41.605582 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.605504    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28aef5e4-132b-450d-84dc-022a68b721c0-kube-api-access-mmf8n" (OuterVolumeSpecName: "kube-api-access-mmf8n") pod "28aef5e4-132b-450d-84dc-022a68b721c0" (UID: "28aef5e4-132b-450d-84dc-022a68b721c0"). InnerVolumeSpecName "kube-api-access-mmf8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:02:41.705216 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.704935    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28aef5e4-132b-450d-84dc-022a68b721c0-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:41.705216 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.704974    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmf8n\" (UniqueName: \"kubernetes.io/projected/28aef5e4-132b-450d-84dc-022a68b721c0-kube-api-access-mmf8n\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:41.705216 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.704991    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28aef5e4-132b-450d-84dc-022a68b721c0-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:41.912492 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:41.912421    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2" podUID="0e93c8f8-7029-4fe3-81cc-3494f883c340" containerName="main" containerID="cri-o://db4e44192af4f183e8f5476b18a840e889d88f73f0a38451260033d342d1840f" gracePeriod=45
Sep 05 03:02:42.153568 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.153541    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:02:42.310356 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.310308    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-home\") pod \"0e93c8f8-7029-4fe3-81cc-3494f883c340\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") "
Sep 05 03:02:42.310356 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.310364    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e93c8f8-7029-4fe3-81cc-3494f883c340-tls-certs\") pod \"0e93c8f8-7029-4fe3-81cc-3494f883c340\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") "
Sep 05 03:02:42.310955 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.310409    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bjgt\" (UniqueName: \"kubernetes.io/projected/0e93c8f8-7029-4fe3-81cc-3494f883c340-kube-api-access-8bjgt\") pod \"0e93c8f8-7029-4fe3-81cc-3494f883c340\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") "
Sep 05 03:02:42.310955 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.310493    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-model-cache\") pod \"0e93c8f8-7029-4fe3-81cc-3494f883c340\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") "
Sep 05 03:02:42.310955 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.310541    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-dshm\") pod \"0e93c8f8-7029-4fe3-81cc-3494f883c340\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") "
Sep 05 03:02:42.310955 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.310564    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-tmp-dir\") pod \"0e93c8f8-7029-4fe3-81cc-3494f883c340\" (UID: \"0e93c8f8-7029-4fe3-81cc-3494f883c340\") "
Sep 05 03:02:42.310955 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.310639    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-home" (OuterVolumeSpecName: "home") pod "0e93c8f8-7029-4fe3-81cc-3494f883c340" (UID: "0e93c8f8-7029-4fe3-81cc-3494f883c340"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:02:42.310955 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.310847    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e93c8f8-7029-4fe3-81cc-3494f883c340" (UID: "0e93c8f8-7029-4fe3-81cc-3494f883c340"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:02:42.311305 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.311114    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e93c8f8-7029-4fe3-81cc-3494f883c340" (UID: "0e93c8f8-7029-4fe3-81cc-3494f883c340"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:02:42.311305 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.311222    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:42.311305 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.311241    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:42.312897 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.312868    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e93c8f8-7029-4fe3-81cc-3494f883c340-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e93c8f8-7029-4fe3-81cc-3494f883c340" (UID: "0e93c8f8-7029-4fe3-81cc-3494f883c340"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:02:42.313245 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.313229    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-dshm" (OuterVolumeSpecName: "dshm") pod "0e93c8f8-7029-4fe3-81cc-3494f883c340" (UID: "0e93c8f8-7029-4fe3-81cc-3494f883c340"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:02:42.313326 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.313253    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e93c8f8-7029-4fe3-81cc-3494f883c340-kube-api-access-8bjgt" (OuterVolumeSpecName: "kube-api-access-8bjgt") pod "0e93c8f8-7029-4fe3-81cc-3494f883c340" (UID: "0e93c8f8-7029-4fe3-81cc-3494f883c340"). InnerVolumeSpecName "kube-api-access-8bjgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:02:42.412311 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.412264    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e93c8f8-7029-4fe3-81cc-3494f883c340-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:42.412311 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.412304    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bjgt\" (UniqueName: \"kubernetes.io/projected/0e93c8f8-7029-4fe3-81cc-3494f883c340-kube-api-access-8bjgt\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:42.412311 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.412316    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:42.412591 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.412325    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e93c8f8-7029-4fe3-81cc-3494f883c340-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:02:42.490679 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.490641    2557 generic.go:356] "Generic (PLEG): container finished" podID="0e93c8f8-7029-4fe3-81cc-3494f883c340" containerID="db4e44192af4f183e8f5476b18a840e889d88f73f0a38451260033d342d1840f" exitCode=0
Sep 05 03:02:42.490886 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.490686    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"
Sep 05 03:02:42.490886 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.490732    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2" event={"ID":"0e93c8f8-7029-4fe3-81cc-3494f883c340","Type":"ContainerDied","Data":"db4e44192af4f183e8f5476b18a840e889d88f73f0a38451260033d342d1840f"}
Sep 05 03:02:42.490886 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.490796    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2" event={"ID":"0e93c8f8-7029-4fe3-81cc-3494f883c340","Type":"ContainerDied","Data":"f3d921aa0ba8759a9cfe41cdc01b15a1ffd0321584b658e628548a5e71259a1a"}
Sep 05 03:02:42.490886 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.490827    2557 scope.go:117] "RemoveContainer" containerID="db4e44192af4f183e8f5476b18a840e889d88f73f0a38451260033d342d1840f"
Sep 05 03:02:42.492341 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.492322    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr" event={"ID":"28aef5e4-132b-450d-84dc-022a68b721c0","Type":"ContainerDied","Data":"b00a288f769e2883e94efb719ab2c2e991576d44b6688f94435ab0b24d2a9bd9"}
Sep 05 03:02:42.492406 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.492333    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"
Sep 05 03:02:42.500924 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.500903    2557 scope.go:117] "RemoveContainer" containerID="db4e44192af4f183e8f5476b18a840e889d88f73f0a38451260033d342d1840f"
Sep 05 03:02:42.501217 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:02:42.501192    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4e44192af4f183e8f5476b18a840e889d88f73f0a38451260033d342d1840f\": container with ID starting with db4e44192af4f183e8f5476b18a840e889d88f73f0a38451260033d342d1840f not found: ID does not exist" containerID="db4e44192af4f183e8f5476b18a840e889d88f73f0a38451260033d342d1840f"
Sep 05 03:02:42.501309 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.501223    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4e44192af4f183e8f5476b18a840e889d88f73f0a38451260033d342d1840f"} err="failed to get container status \"db4e44192af4f183e8f5476b18a840e889d88f73f0a38451260033d342d1840f\": rpc error: code = NotFound desc = could not find container \"db4e44192af4f183e8f5476b18a840e889d88f73f0a38451260033d342d1840f\": container with ID starting with db4e44192af4f183e8f5476b18a840e889d88f73f0a38451260033d342d1840f not found: ID does not exist"
Sep 05 03:02:42.501309 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.501239    2557 scope.go:117] "RemoveContainer" containerID="511f87990bd7eb96aec7eb216a2517210177a1ba237aedd5c3e3d968fdecbc03"
Sep 05 03:02:42.513075 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.513048    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"]
Sep 05 03:02:42.519402 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.519341    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-ht5f2"]
Sep 05 03:02:42.535635 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.535598    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"]
Sep 05 03:02:42.538342 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.538313    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-g6mgr"]
Sep 05 03:02:42.991277 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.991239    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e93c8f8-7029-4fe3-81cc-3494f883c340" path="/var/lib/kubelet/pods/0e93c8f8-7029-4fe3-81cc-3494f883c340/volumes"
Sep 05 03:02:42.991801 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:42.991783    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28aef5e4-132b-450d-84dc-022a68b721c0" path="/var/lib/kubelet/pods/28aef5e4-132b-450d-84dc-022a68b721c0/volumes"
Sep 05 03:02:44.743494 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:44.743458    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:44.757208 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:44.757174    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:02:45.523161 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:02:45.523117    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:03:01.412360 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.412325    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"]
Sep 05 03:03:01.413028 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.412713    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28aef5e4-132b-450d-84dc-022a68b721c0" containerName="main"
Sep 05 03:03:01.413028 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.412727    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="28aef5e4-132b-450d-84dc-022a68b721c0" containerName="main"
Sep 05 03:03:01.413028 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.412737    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e93c8f8-7029-4fe3-81cc-3494f883c340" containerName="main"
Sep 05 03:03:01.413028 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.412744    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e93c8f8-7029-4fe3-81cc-3494f883c340" containerName="main"
Sep 05 03:03:01.413028 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.412804    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e93c8f8-7029-4fe3-81cc-3494f883c340" containerName="main"
Sep 05 03:03:01.413028 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.412815    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="28aef5e4-132b-450d-84dc-022a68b721c0" containerName="main"
Sep 05 03:03:01.416040 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.416020    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.418104 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.418084    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Sep 05 03:03:01.425865 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.425840    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"]
Sep 05 03:03:01.445410 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.445382    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:03:01.472662 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.472630    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-model-cache\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.472780 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.472675    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-home\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.472780 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.472754    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.472871 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.472783    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5efd63a-7185-4a84-9a94-6413af61e9d5-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.472871 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.472814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgksn\" (UniqueName: \"kubernetes.io/projected/b5efd63a-7185-4a84-9a94-6413af61e9d5-kube-api-access-hgksn\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.472871 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.472854    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-dshm\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.573534 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.573499    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgksn\" (UniqueName: \"kubernetes.io/projected/b5efd63a-7185-4a84-9a94-6413af61e9d5-kube-api-access-hgksn\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.573692 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.573550    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-dshm\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.573692 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.573606    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-model-cache\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.573692 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.573636    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-home\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.573692 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.573674    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.573959 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.573697    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5efd63a-7185-4a84-9a94-6413af61e9d5-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.574200 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.574149    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.579678 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.574592    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-model-cache\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.579678 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.574694    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-home\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.579678 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.576056    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-dshm\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.583396 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.582390    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgksn\" (UniqueName: \"kubernetes.io/projected/b5efd63a-7185-4a84-9a94-6413af61e9d5-kube-api-access-hgksn\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.584172 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.584150    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5efd63a-7185-4a84-9a94-6413af61e9d5-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-bdk7p\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:01.727742 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:01.727652    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:02.000037 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:02.000009    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"]
Sep 05 03:03:02.002360 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:03:02.002328    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5efd63a_7185_4a84_9a94_6413af61e9d5.slice/crio-86fd403cc0ca9c7582c63eadf229d3ea6c60fd875bc64a4737d0725156265327 WatchSource:0}: Error finding container 86fd403cc0ca9c7582c63eadf229d3ea6c60fd875bc64a4737d0725156265327: Status 404 returned error can't find the container with id 86fd403cc0ca9c7582c63eadf229d3ea6c60fd875bc64a4737d0725156265327
Sep 05 03:03:02.603860 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:02.603823    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p" event={"ID":"b5efd63a-7185-4a84-9a94-6413af61e9d5","Type":"ContainerStarted","Data":"0273ee5a1eecb95b89668c8b1d8d63906cee60ee1709484393cb92d1a3d4f678"}
Sep 05 03:03:02.604333 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:02.603878    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p" event={"ID":"b5efd63a-7185-4a84-9a94-6413af61e9d5","Type":"ContainerStarted","Data":"86fd403cc0ca9c7582c63eadf229d3ea6c60fd875bc64a4737d0725156265327"}
Sep 05 03:03:02.620350 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:02.620299    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p" podStartSLOduration=1.6202845670000001 podStartE2EDuration="1.620284567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:03:02.618960244 +0000 UTC m=+2686.174116553" watchObservedRunningTime="2026-09-05 03:03:02.620284567 +0000 UTC m=+2686.175440876"
Sep 05 03:03:02.728357 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:02.728320    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:07.283064 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:07.283029    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:03:11.728751 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:11.728665    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:11.741279 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:11.741251    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:12.672880 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:12.672846    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:03:13.410182 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:13.410144    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:03:13.947822 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:13.947773    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:03:17.167126 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:17.167096    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:03:17.175500 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:17.175472    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:03:33.435804 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:33.435764    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:03:33.788270 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:33.788177    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:03:47.215865 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:47.215822    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"]
Sep 05 03:03:47.245374 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:03:47.245305    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:02.236925 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.236869    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk" podUID="71a78665-9c31-4b02-bb9c-992ffcba6fe3" containerName="main" containerID="cri-o://badc5ac020e22d88c79ca8e0018dfd659c5b57effa76e5843ccd1c6f9c603faf" gracePeriod=45
Sep 05 03:04:02.499733 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.499669    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:04:02.613665 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.613631    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrnvw\" (UniqueName: \"kubernetes.io/projected/71a78665-9c31-4b02-bb9c-992ffcba6fe3-kube-api-access-lrnvw\") pod \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") "
Sep 05 03:04:02.613665 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.613671    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-model-cache\") pod \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") "
Sep 05 03:04:02.613931 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.613697    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-tmp-dir\") pod \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") "
Sep 05 03:04:02.613931 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.613718    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-home\") pod \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") "
Sep 05 03:04:02.613931 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.613737    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-dshm\") pod \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") "
Sep 05 03:04:02.613931 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.613765    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a78665-9c31-4b02-bb9c-992ffcba6fe3-tls-certs\") pod \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\" (UID: \"71a78665-9c31-4b02-bb9c-992ffcba6fe3\") "
Sep 05 03:04:02.614126 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.614026    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-model-cache" (OuterVolumeSpecName: "model-cache") pod "71a78665-9c31-4b02-bb9c-992ffcba6fe3" (UID: "71a78665-9c31-4b02-bb9c-992ffcba6fe3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:04:02.614126 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.614038    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71a78665-9c31-4b02-bb9c-992ffcba6fe3" (UID: "71a78665-9c31-4b02-bb9c-992ffcba6fe3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:04:02.614126 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.614055    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-home" (OuterVolumeSpecName: "home") pod "71a78665-9c31-4b02-bb9c-992ffcba6fe3" (UID: "71a78665-9c31-4b02-bb9c-992ffcba6fe3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:04:02.615958 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.615927    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a78665-9c31-4b02-bb9c-992ffcba6fe3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71a78665-9c31-4b02-bb9c-992ffcba6fe3" (UID: "71a78665-9c31-4b02-bb9c-992ffcba6fe3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:04:02.615958 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.615951    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-dshm" (OuterVolumeSpecName: "dshm") pod "71a78665-9c31-4b02-bb9c-992ffcba6fe3" (UID: "71a78665-9c31-4b02-bb9c-992ffcba6fe3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:04:02.616108 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.616026    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a78665-9c31-4b02-bb9c-992ffcba6fe3-kube-api-access-lrnvw" (OuterVolumeSpecName: "kube-api-access-lrnvw") pod "71a78665-9c31-4b02-bb9c-992ffcba6fe3" (UID: "71a78665-9c31-4b02-bb9c-992ffcba6fe3"). InnerVolumeSpecName "kube-api-access-lrnvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:04:02.715352 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.715297    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:04:02.715352 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.715343    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:04:02.715352 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.715351    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:04:02.715352 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.715359    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a78665-9c31-4b02-bb9c-992ffcba6fe3-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:04:02.715352 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.715369    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrnvw\" (UniqueName: \"kubernetes.io/projected/71a78665-9c31-4b02-bb9c-992ffcba6fe3-kube-api-access-lrnvw\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:04:02.715686 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.715378    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a78665-9c31-4b02-bb9c-992ffcba6fe3-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:04:02.934239 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.934198    2557 generic.go:356] "Generic (PLEG): container finished" podID="71a78665-9c31-4b02-bb9c-992ffcba6fe3" containerID="badc5ac020e22d88c79ca8e0018dfd659c5b57effa76e5843ccd1c6f9c603faf" exitCode=0
Sep 05 03:04:02.934413 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.934245    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"
Sep 05 03:04:02.934413 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.934279    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk" event={"ID":"71a78665-9c31-4b02-bb9c-992ffcba6fe3","Type":"ContainerDied","Data":"badc5ac020e22d88c79ca8e0018dfd659c5b57effa76e5843ccd1c6f9c603faf"}
Sep 05 03:04:02.934413 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.934338    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk" event={"ID":"71a78665-9c31-4b02-bb9c-992ffcba6fe3","Type":"ContainerDied","Data":"6c63b8e07e27016d976a8f71a18f31703009df8c87c8e56e6a14600b44b95836"}
Sep 05 03:04:02.934413 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.934362    2557 scope.go:117] "RemoveContainer" containerID="badc5ac020e22d88c79ca8e0018dfd659c5b57effa76e5843ccd1c6f9c603faf"
Sep 05 03:04:02.943746 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.943614    2557 scope.go:117] "RemoveContainer" containerID="badc5ac020e22d88c79ca8e0018dfd659c5b57effa76e5843ccd1c6f9c603faf"
Sep 05 03:04:02.943975 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:04:02.943944    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"badc5ac020e22d88c79ca8e0018dfd659c5b57effa76e5843ccd1c6f9c603faf\": container with ID starting with badc5ac020e22d88c79ca8e0018dfd659c5b57effa76e5843ccd1c6f9c603faf not found: ID does not exist" containerID="badc5ac020e22d88c79ca8e0018dfd659c5b57effa76e5843ccd1c6f9c603faf"
Sep 05 03:04:02.944087 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.943982    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"badc5ac020e22d88c79ca8e0018dfd659c5b57effa76e5843ccd1c6f9c603faf"} err="failed to get container status \"badc5ac020e22d88c79ca8e0018dfd659c5b57effa76e5843ccd1c6f9c603faf\": rpc error: code = NotFound desc = could not find container \"badc5ac020e22d88c79ca8e0018dfd659c5b57effa76e5843ccd1c6f9c603faf\": container with ID starting with badc5ac020e22d88c79ca8e0018dfd659c5b57effa76e5843ccd1c6f9c603faf not found: ID does not exist"
Sep 05 03:04:02.957548 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.957509    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"]
Sep 05 03:04:02.963170 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.963139    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qk5xk"]
Sep 05 03:04:02.991770 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:02.991739    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71a78665-9c31-4b02-bb9c-992ffcba6fe3" path="/var/lib/kubelet/pods/71a78665-9c31-4b02-bb9c-992ffcba6fe3/volumes"
Sep 05 03:04:08.693658 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.693611    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:08.790424 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.790387    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:08.933644 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.933606    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"]
Sep 05 03:04:08.934055 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.934040    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a78665-9c31-4b02-bb9c-992ffcba6fe3" containerName="main"
Sep 05 03:04:08.934101 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.934058    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a78665-9c31-4b02-bb9c-992ffcba6fe3" containerName="main"
Sep 05 03:04:08.934137 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.934119    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a78665-9c31-4b02-bb9c-992ffcba6fe3" containerName="main"
Sep 05 03:04:08.939236 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.939211    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:08.942371 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.942315    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 05 03:04:08.942834 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.942810    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 05 03:04:08.942948 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.942847    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 05 03:04:08.943040 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.943019    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-hbc74\""
Sep 05 03:04:08.952920 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.952893    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"]
Sep 05 03:04:08.968359 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.968330    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:08.968538 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.968386    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:08.968538 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.968495    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mph6s\" (UniqueName: \"kubernetes.io/projected/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-kube-api-access-mph6s\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:08.968669 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.968568    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:08.968669 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.968599    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:08.968669 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.968624    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:08.968794 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.968700    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:08.968794 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.968750    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:08.968794 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:08.968787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.018187 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.018150    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:09.069795 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.069757    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.069795 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.069798    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mph6s\" (UniqueName: \"kubernetes.io/projected/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-kube-api-access-mph6s\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.070065 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.069932    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.070065 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.069973    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.070065 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.069994    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.070065 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.070024    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.070284 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.070068    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.070284 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.070101    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.070284 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.070199    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.070465 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.070334    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.070465 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.070384    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.070596 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.070512    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.070596 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.070554    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.070706 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.070685    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.072515 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.072493    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.072740 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.072719    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.079236 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.079209    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.079378 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.079351    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mph6s\" (UniqueName: \"kubernetes.io/projected/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-kube-api-access-mph6s\") pod \"router-gateway-1-openshift-default-55f8b478f9-vvgx2\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.121638 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.121600    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:09.251293 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.251200    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:09.555217 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.551283    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"]
Sep 05 03:04:09.557913 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.557858    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:04:09.558041 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.557935    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:04:09.558041 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.557966    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:04:09.975586 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.975550    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2" event={"ID":"9cc72fc2-6137-4ab5-a137-3b8621d3fd02","Type":"ContainerStarted","Data":"ade50167a62903fa180c598aaaada0ce809a0be580e7a1e8f1eb374b5319a3f9"}
Sep 05 03:04:09.975956 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:09.975605    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2" event={"ID":"9cc72fc2-6137-4ab5-a137-3b8621d3fd02","Type":"ContainerStarted","Data":"bb3b7e807214cc0b1895e396465e08089777d049dcc475c433f892011f845c3a"}
Sep 05 03:04:10.005226 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:10.005161    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2" podStartSLOduration=2.005139816 podStartE2EDuration="2.005139816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:04:10.004351251 +0000 UTC m=+2753.559507559" watchObservedRunningTime="2026-09-05 03:04:10.005139816 +0000 UTC m=+2753.560296126"
Sep 05 03:04:10.251765 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:10.251671    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:10.256600 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:10.256572    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:10.980809 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:10.980770    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:10.981894 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:10.981876    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:12.439700 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:12.439647    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:13.652411 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:13.652366    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"]
Sep 05 03:04:13.748640 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:13.748608    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:18.529041 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:18.529009    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:20.390132 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:20.390103    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:20.525328 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:20.525287    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:21.621264 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:21.621220    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:25.195792 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:25.195756    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:26.868500 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:26.868456    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:28.740927 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:28.740847    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p" podUID="b5efd63a-7185-4a84-9a94-6413af61e9d5" containerName="main" containerID="cri-o://0273ee5a1eecb95b89668c8b1d8d63906cee60ee1709484393cb92d1a3d4f678" gracePeriod=45
Sep 05 03:04:28.994227 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:28.994159    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:04:29.088038 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.087995    2557 generic.go:356] "Generic (PLEG): container finished" podID="b5efd63a-7185-4a84-9a94-6413af61e9d5" containerID="0273ee5a1eecb95b89668c8b1d8d63906cee60ee1709484393cb92d1a3d4f678" exitCode=0
Sep 05 03:04:29.088210 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.088040    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p" event={"ID":"b5efd63a-7185-4a84-9a94-6413af61e9d5","Type":"ContainerDied","Data":"0273ee5a1eecb95b89668c8b1d8d63906cee60ee1709484393cb92d1a3d4f678"}
Sep 05 03:04:29.088210 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.088101    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p" event={"ID":"b5efd63a-7185-4a84-9a94-6413af61e9d5","Type":"ContainerDied","Data":"86fd403cc0ca9c7582c63eadf229d3ea6c60fd875bc64a4737d0725156265327"}
Sep 05 03:04:29.088210 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.088124    2557 scope.go:117] "RemoveContainer" containerID="0273ee5a1eecb95b89668c8b1d8d63906cee60ee1709484393cb92d1a3d4f678"
Sep 05 03:04:29.088210 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.088049    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"
Sep 05 03:04:29.098386 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.098366    2557 scope.go:117] "RemoveContainer" containerID="0273ee5a1eecb95b89668c8b1d8d63906cee60ee1709484393cb92d1a3d4f678"
Sep 05 03:04:29.098791 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:04:29.098760    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0273ee5a1eecb95b89668c8b1d8d63906cee60ee1709484393cb92d1a3d4f678\": container with ID starting with 0273ee5a1eecb95b89668c8b1d8d63906cee60ee1709484393cb92d1a3d4f678 not found: ID does not exist" containerID="0273ee5a1eecb95b89668c8b1d8d63906cee60ee1709484393cb92d1a3d4f678"
Sep 05 03:04:29.098870 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.098826    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0273ee5a1eecb95b89668c8b1d8d63906cee60ee1709484393cb92d1a3d4f678"} err="failed to get container status \"0273ee5a1eecb95b89668c8b1d8d63906cee60ee1709484393cb92d1a3d4f678\": rpc error: code = NotFound desc = could not find container \"0273ee5a1eecb95b89668c8b1d8d63906cee60ee1709484393cb92d1a3d4f678\": container with ID starting with 0273ee5a1eecb95b89668c8b1d8d63906cee60ee1709484393cb92d1a3d4f678 not found: ID does not exist"
Sep 05 03:04:29.157201 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.157162    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-model-cache\") pod \"b5efd63a-7185-4a84-9a94-6413af61e9d5\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") "
Sep 05 03:04:29.157372 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.157221    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgksn\" (UniqueName: \"kubernetes.io/projected/b5efd63a-7185-4a84-9a94-6413af61e9d5-kube-api-access-hgksn\") pod \"b5efd63a-7185-4a84-9a94-6413af61e9d5\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") "
Sep 05 03:04:29.157372 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.157258    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-home\") pod \"b5efd63a-7185-4a84-9a94-6413af61e9d5\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") "
Sep 05 03:04:29.157372 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.157300    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5efd63a-7185-4a84-9a94-6413af61e9d5-tls-certs\") pod \"b5efd63a-7185-4a84-9a94-6413af61e9d5\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") "
Sep 05 03:04:29.157372 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.157343    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-dshm\") pod \"b5efd63a-7185-4a84-9a94-6413af61e9d5\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") "
Sep 05 03:04:29.157639 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.157410    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-tmp-dir\") pod \"b5efd63a-7185-4a84-9a94-6413af61e9d5\" (UID: \"b5efd63a-7185-4a84-9a94-6413af61e9d5\") "
Sep 05 03:04:29.157639 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.157532    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-home" (OuterVolumeSpecName: "home") pod "b5efd63a-7185-4a84-9a94-6413af61e9d5" (UID: "b5efd63a-7185-4a84-9a94-6413af61e9d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:04:29.157639 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.157541    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5efd63a-7185-4a84-9a94-6413af61e9d5" (UID: "b5efd63a-7185-4a84-9a94-6413af61e9d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:04:29.157984 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.157944    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5efd63a-7185-4a84-9a94-6413af61e9d5" (UID: "b5efd63a-7185-4a84-9a94-6413af61e9d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:04:29.158654 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.158631    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:04:29.158757 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.158656    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:04:29.158757 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.158670    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:04:29.159705 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.159670    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-dshm" (OuterVolumeSpecName: "dshm") pod "b5efd63a-7185-4a84-9a94-6413af61e9d5" (UID: "b5efd63a-7185-4a84-9a94-6413af61e9d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:04:29.159856 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.159759    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5efd63a-7185-4a84-9a94-6413af61e9d5-kube-api-access-hgksn" (OuterVolumeSpecName: "kube-api-access-hgksn") pod "b5efd63a-7185-4a84-9a94-6413af61e9d5" (UID: "b5efd63a-7185-4a84-9a94-6413af61e9d5"). InnerVolumeSpecName "kube-api-access-hgksn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:04:29.159922 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.159880    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5efd63a-7185-4a84-9a94-6413af61e9d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5efd63a-7185-4a84-9a94-6413af61e9d5" (UID: "b5efd63a-7185-4a84-9a94-6413af61e9d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:04:29.259210 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.259114    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgksn\" (UniqueName: \"kubernetes.io/projected/b5efd63a-7185-4a84-9a94-6413af61e9d5-kube-api-access-hgksn\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:04:29.259210 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.259156    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5efd63a-7185-4a84-9a94-6413af61e9d5-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:04:29.259210 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.259167    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5efd63a-7185-4a84-9a94-6413af61e9d5-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:04:29.415587 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.415551    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"]
Sep 05 03:04:29.420798 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:29.420765    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-bdk7p"]
Sep 05 03:04:30.992262 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:30.992225    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5efd63a-7185-4a84-9a94-6413af61e9d5" path="/var/lib/kubelet/pods/b5efd63a-7185-4a84-9a94-6413af61e9d5/volumes"
Sep 05 03:04:31.099929 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.099892    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"]
Sep 05 03:04:31.100288 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.100272    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5efd63a-7185-4a84-9a94-6413af61e9d5" containerName="main"
Sep 05 03:04:31.100357 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.100290    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5efd63a-7185-4a84-9a94-6413af61e9d5" containerName="main"
Sep 05 03:04:31.100413 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.100378    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5efd63a-7185-4a84-9a94-6413af61e9d5" containerName="main"
Sep 05 03:04:31.102763 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.102741    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.104614 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.104589    2557 generic.go:356] "Generic (PLEG): container finished" podID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerID="ade50167a62903fa180c598aaaada0ce809a0be580e7a1e8f1eb374b5319a3f9" exitCode=0
Sep 05 03:04:31.104739 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.104664    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2" event={"ID":"9cc72fc2-6137-4ab5-a137-3b8621d3fd02","Type":"ContainerDied","Data":"ade50167a62903fa180c598aaaada0ce809a0be580e7a1e8f1eb374b5319a3f9"}
Sep 05 03:04:31.105102 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.105083    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 05 03:04:31.105197 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.105168    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 05 03:04:31.105197 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.105172    2557 scope.go:117] "RemoveContainer" containerID="ade50167a62903fa180c598aaaada0ce809a0be580e7a1e8f1eb374b5319a3f9"
Sep 05 03:04:31.106147 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.106122    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 05 03:04:31.106247 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.106149    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:04:31.106247 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.106206    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-bz4ww\""
Sep 05 03:04:31.106370 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.106251    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:04:31.106370 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.106303    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:04:31.115159 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.115126    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"]
Sep 05 03:04:31.262829 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.262724    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:31.281927 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.281897    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.282081 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.281932    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.282081 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.281966    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7smhf\" (UniqueName: \"kubernetes.io/projected/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-kube-api-access-7smhf\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.282081 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.282050    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.282216 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.282096    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.282216 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.282198    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.383847 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.383802    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.384019 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.383854    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.384019 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.383890    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7smhf\" (UniqueName: \"kubernetes.io/projected/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-kube-api-access-7smhf\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.384019 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.383925    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.384019 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.383958    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.384019 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.383999    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.384406 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.384281    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.384406 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.384354    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.384552 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.384433    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.386248 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.386220    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.386878 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.386843    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.407470 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.407431    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7smhf\" (UniqueName: \"kubernetes.io/projected/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-kube-api-access-7smhf\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.433299 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.433256    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:31.459989 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.459955    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:31.740661 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:31.740300    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"]
Sep 05 03:04:31.743046 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:04:31.742991    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb55ab6_97a3_416b_b1d5_7f8f3c0c5e30.slice/crio-fe64c820e885ebc0c52c617e2efd93c642189c407c5cf05a7fbd89ba8ff418d7 WatchSource:0}: Error finding container fe64c820e885ebc0c52c617e2efd93c642189c407c5cf05a7fbd89ba8ff418d7: Status 404 returned error can't find the container with id fe64c820e885ebc0c52c617e2efd93c642189c407c5cf05a7fbd89ba8ff418d7
Sep 05 03:04:32.112123 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:32.112079    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc" event={"ID":"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30","Type":"ContainerStarted","Data":"4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39"}
Sep 05 03:04:32.112621 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:32.112150    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc" event={"ID":"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30","Type":"ContainerStarted","Data":"fe64c820e885ebc0c52c617e2efd93c642189c407c5cf05a7fbd89ba8ff418d7"}
Sep 05 03:04:32.114386 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:32.114358    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2" event={"ID":"9cc72fc2-6137-4ab5-a137-3b8621d3fd02","Type":"ContainerStarted","Data":"e88ac63fb87abcdd2b189143b6b2b5d2e5b41ddcbed915dc8000dac2c7a43a6c"}
Sep 05 03:04:32.133246 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:32.133192    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc" podStartSLOduration=1.133176743 podStartE2EDuration="1.133176743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:04:32.132414545 +0000 UTC m=+2775.687570845" watchObservedRunningTime="2026-09-05 03:04:32.133176743 +0000 UTC m=+2775.688333052"
Sep 05 03:04:32.251452 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:32.251382    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:32.251911 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:32.251862    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2" podUID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 05 03:04:32.460735 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:32.460576    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:33.105723 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:33.105688    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:33.228565 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:33.228524    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:33.252727 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:33.252676    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2" podUID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 05 03:04:34.251923 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:34.251879    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2" podUID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 05 03:04:35.252583 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:35.252535    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2" podUID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 05 03:04:36.252604 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:36.252554    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2" podUID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 05 03:04:37.252485 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:37.252423    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2" podUID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 05 03:04:38.255786 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:38.255753    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:38.256163 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:38.255991    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:38.256853 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:38.256832    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:04:41.460516 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:41.460401    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:41.473668 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:41.473638    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:42.183068 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:42.183029    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:04:49.327988 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:49.327945    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:50.955163 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:50.955125    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:56.705363 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:56.705327    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:04:56.877917 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:04:56.877878    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:05:10.412818 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.412777    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"]
Sep 05 03:05:10.416208 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.416185    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.419168 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.419139    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 05 03:05:10.424717 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.424689    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"]
Sep 05 03:05:10.463592 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.463559    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:05:10.522590 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.522549    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrvff\" (UniqueName: \"kubernetes.io/projected/386ef6c5-792a-44be-a8fa-aa073b4f361f-kube-api-access-nrvff\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.522756 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.522596    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.522756 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.522671    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.522756 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.522731    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.522870 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.522757    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.522870 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.522775    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386ef6c5-792a-44be-a8fa-aa073b4f361f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.623810 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.623768    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386ef6c5-792a-44be-a8fa-aa073b4f361f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.624005 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.623874    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrvff\" (UniqueName: \"kubernetes.io/projected/386ef6c5-792a-44be-a8fa-aa073b4f361f-kube-api-access-nrvff\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.624005 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.623908    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.624005 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.623963    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.624176 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.624016    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.624176 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.624044    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.624410 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.624383    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.624519 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.624421    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.624519 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.624510    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.626985 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.626955    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.627143 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.627124    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386ef6c5-792a-44be-a8fa-aa073b4f361f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.637973 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.637942    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrvff\" (UniqueName: \"kubernetes.io/projected/386ef6c5-792a-44be-a8fa-aa073b4f361f-kube-api-access-nrvff\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:10.669553 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.669468    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:05:10.728620 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:10.728583    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:11.016367 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:11.011398    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"]
Sep 05 03:05:11.328498 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:11.328457    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h" event={"ID":"386ef6c5-792a-44be-a8fa-aa073b4f361f","Type":"ContainerStarted","Data":"6fd0b0e1f80f5ed511057d5e4ab17b912887be984d144349c9ae276700dd8d3f"}
Sep 05 03:05:11.328677 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:11.328517    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h" event={"ID":"386ef6c5-792a-44be-a8fa-aa073b4f361f","Type":"ContainerStarted","Data":"af7d04bb4423a44491f5dfe1389de2ffc6dfa9d0818f4b9a7176abb6891ca611"}
Sep 05 03:05:11.347922 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:11.347863    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h" podStartSLOduration=1.34784521 podStartE2EDuration="1.34784521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:05:11.346399071 +0000 UTC m=+2814.901555381" watchObservedRunningTime="2026-09-05 03:05:11.34784521 +0000 UTC m=+2814.903001550"
Sep 05 03:05:11.729004 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:11.728917    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:20.728737 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:20.728681    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:20.741707 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:20.741661    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:21.034135 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:21.034031    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:05:21.396121 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:21.396077    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:05:37.430548 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:37.430503    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:05:42.324699 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:42.324657    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:05:42.425861 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:42.425815    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:05:47.394252 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:47.394214    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"]
Sep 05 03:05:47.394691 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:47.394414    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2" podUID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerName="istio-proxy" containerID="cri-o://e88ac63fb87abcdd2b189143b6b2b5d2e5b41ddcbed915dc8000dac2c7a43a6c" gracePeriod=30
Sep 05 03:05:47.532126 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:47.532089    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:05:52.553933 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.553898    2557 generic.go:356] "Generic (PLEG): container finished" podID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerID="e88ac63fb87abcdd2b189143b6b2b5d2e5b41ddcbed915dc8000dac2c7a43a6c" exitCode=0
Sep 05 03:05:52.554371 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.553973    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2" event={"ID":"9cc72fc2-6137-4ab5-a137-3b8621d3fd02","Type":"ContainerDied","Data":"e88ac63fb87abcdd2b189143b6b2b5d2e5b41ddcbed915dc8000dac2c7a43a6c"}
Sep 05 03:05:52.554371 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.554098    2557 scope.go:117] "RemoveContainer" containerID="ade50167a62903fa180c598aaaada0ce809a0be580e7a1e8f1eb374b5319a3f9"
Sep 05 03:05:52.652055 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.652027    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:05:52.711491 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.711453    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-token\") pod \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") "
Sep 05 03:05:52.711675 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.711554    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-data\") pod \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") "
Sep 05 03:05:52.711675 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.711583    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-podinfo\") pod \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") "
Sep 05 03:05:52.711675 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.711627    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-credential-socket\") pod \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") "
Sep 05 03:05:52.711858 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.711678    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-envoy\") pod \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") "
Sep 05 03:05:52.711858 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.711762    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-workload-socket\") pod \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") "
Sep 05 03:05:52.711858 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.711787    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istiod-ca-cert\") pod \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") "
Sep 05 03:05:52.711858 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.711813    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mph6s\" (UniqueName: \"kubernetes.io/projected/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-kube-api-access-mph6s\") pod \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") "
Sep 05 03:05:52.711858 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.711857    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-workload-certs\") pod \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\" (UID: \"9cc72fc2-6137-4ab5-a137-3b8621d3fd02\") "
Sep 05 03:05:52.712129 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.711990    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9cc72fc2-6137-4ab5-a137-3b8621d3fd02" (UID: "9cc72fc2-6137-4ab5-a137-3b8621d3fd02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:05:52.712189 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.712171    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-credential-socket\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:05:52.712276 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.712249    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-data" (OuterVolumeSpecName: "istio-data") pod "9cc72fc2-6137-4ab5-a137-3b8621d3fd02" (UID: "9cc72fc2-6137-4ab5-a137-3b8621d3fd02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:05:52.712340 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.712290    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9cc72fc2-6137-4ab5-a137-3b8621d3fd02" (UID: "9cc72fc2-6137-4ab5-a137-3b8621d3fd02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:05:52.712616 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.712562    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9cc72fc2-6137-4ab5-a137-3b8621d3fd02" (UID: "9cc72fc2-6137-4ab5-a137-3b8621d3fd02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:05:52.712817 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.712754    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9cc72fc2-6137-4ab5-a137-3b8621d3fd02" (UID: "9cc72fc2-6137-4ab5-a137-3b8621d3fd02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:05:52.714631 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.714591    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-token" (OuterVolumeSpecName: "istio-token") pod "9cc72fc2-6137-4ab5-a137-3b8621d3fd02" (UID: "9cc72fc2-6137-4ab5-a137-3b8621d3fd02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:05:52.714763 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.714736    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-kube-api-access-mph6s" (OuterVolumeSpecName: "kube-api-access-mph6s") pod "9cc72fc2-6137-4ab5-a137-3b8621d3fd02" (UID: "9cc72fc2-6137-4ab5-a137-3b8621d3fd02"). InnerVolumeSpecName "kube-api-access-mph6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:05:52.714853 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.714802    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9cc72fc2-6137-4ab5-a137-3b8621d3fd02" (UID: "9cc72fc2-6137-4ab5-a137-3b8621d3fd02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:05:52.714912 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.714886    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9cc72fc2-6137-4ab5-a137-3b8621d3fd02" (UID: "9cc72fc2-6137-4ab5-a137-3b8621d3fd02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 03:05:52.813100 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.813054    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-data\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:05:52.813100 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.813092    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-podinfo\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:05:52.813100 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.813106    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-envoy\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:05:52.813363 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.813118    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-workload-socket\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:05:52.813363 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.813132    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istiod-ca-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:05:52.813363 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.813143    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mph6s\" (UniqueName: \"kubernetes.io/projected/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-kube-api-access-mph6s\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:05:52.813363 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.813155    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-workload-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:05:52.813363 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:52.813166    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc72fc2-6137-4ab5-a137-3b8621d3fd02-istio-token\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:05:53.560369 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:53.560333    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"
Sep 05 03:05:53.560811 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:53.560381    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2" event={"ID":"9cc72fc2-6137-4ab5-a137-3b8621d3fd02","Type":"ContainerDied","Data":"bb3b7e807214cc0b1895e396465e08089777d049dcc475c433f892011f845c3a"}
Sep 05 03:05:53.560811 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:53.560470    2557 scope.go:117] "RemoveContainer" containerID="e88ac63fb87abcdd2b189143b6b2b5d2e5b41ddcbed915dc8000dac2c7a43a6c"
Sep 05 03:05:53.578086 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:53.578058    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"]
Sep 05 03:05:53.582132 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:53.582108    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-vvgx2"]
Sep 05 03:05:54.990779 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:05:54.990743    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" path="/var/lib/kubelet/pods/9cc72fc2-6137-4ab5-a137-3b8621d3fd02/volumes"
Sep 05 03:06:01.308667 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:01.308627    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:06:03.254097 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:03.254064    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:06:08.984780 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:08.984691    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:06:10.279700 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:10.279642    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:06:16.109945 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:16.109912    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:06:16.649577 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:16.649543    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:06:18.425918 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.425876    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:06:18.426352 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.426031    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"]
Sep 05 03:06:18.435397 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:06:18.435353    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 03:06:18.435397 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:06:18.435397 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 03:06:18.435397 ip-10-0-143-128 kubenswrapper[2557]:  > containerID="6fd0b0e1f80f5ed511057d5e4ab17b912887be984d144349c9ae276700dd8d3f" cmd=["/bin/sleep","15"]
Sep 05 03:06:18.435707 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.435415    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h" message="" err=<
Sep 05 03:06:18.435707 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:06:18.435707 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 03:06:18.435707 ip-10-0-143-128 kubenswrapper[2557]:  >
Sep 05 03:06:18.435707 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:06:18.435432    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 03:06:18.435707 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:06:18.435707 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 03:06:18.435707 ip-10-0-143-128 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h" podUID="386ef6c5-792a-44be-a8fa-aa073b4f361f" containerName="main" containerID="cri-o://6fd0b0e1f80f5ed511057d5e4ab17b912887be984d144349c9ae276700dd8d3f"
Sep 05 03:06:18.435707 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.435487    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h" podUID="386ef6c5-792a-44be-a8fa-aa073b4f361f" containerName="main" containerID="cri-o://6fd0b0e1f80f5ed511057d5e4ab17b912887be984d144349c9ae276700dd8d3f" gracePeriod=60
Sep 05 03:06:18.525499 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.525462    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:06:18.709017 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.708979    2557 generic.go:356] "Generic (PLEG): container finished" podID="386ef6c5-792a-44be-a8fa-aa073b4f361f" containerID="6fd0b0e1f80f5ed511057d5e4ab17b912887be984d144349c9ae276700dd8d3f" exitCode=0
Sep 05 03:06:18.709168 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.709048    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h" event={"ID":"386ef6c5-792a-44be-a8fa-aa073b4f361f","Type":"ContainerDied","Data":"6fd0b0e1f80f5ed511057d5e4ab17b912887be984d144349c9ae276700dd8d3f"}
Sep 05 03:06:18.795232 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.795204    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:06:18.850984 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.850952    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386ef6c5-792a-44be-a8fa-aa073b4f361f-tls-certs\") pod \"386ef6c5-792a-44be-a8fa-aa073b4f361f\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") "
Sep 05 03:06:18.851168 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.850989    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrvff\" (UniqueName: \"kubernetes.io/projected/386ef6c5-792a-44be-a8fa-aa073b4f361f-kube-api-access-nrvff\") pod \"386ef6c5-792a-44be-a8fa-aa073b4f361f\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") "
Sep 05 03:06:18.851168 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.851013    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-model-cache\") pod \"386ef6c5-792a-44be-a8fa-aa073b4f361f\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") "
Sep 05 03:06:18.851168 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.851029    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-home\") pod \"386ef6c5-792a-44be-a8fa-aa073b4f361f\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") "
Sep 05 03:06:18.851168 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.851104    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-tmp-dir\") pod \"386ef6c5-792a-44be-a8fa-aa073b4f361f\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") "
Sep 05 03:06:18.851168 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.851132    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-dshm\") pod \"386ef6c5-792a-44be-a8fa-aa073b4f361f\" (UID: \"386ef6c5-792a-44be-a8fa-aa073b4f361f\") "
Sep 05 03:06:18.851453 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.851248    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-model-cache" (OuterVolumeSpecName: "model-cache") pod "386ef6c5-792a-44be-a8fa-aa073b4f361f" (UID: "386ef6c5-792a-44be-a8fa-aa073b4f361f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:06:18.851453 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.851320    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-home" (OuterVolumeSpecName: "home") pod "386ef6c5-792a-44be-a8fa-aa073b4f361f" (UID: "386ef6c5-792a-44be-a8fa-aa073b4f361f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:06:18.851540 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.851482    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "386ef6c5-792a-44be-a8fa-aa073b4f361f" (UID: "386ef6c5-792a-44be-a8fa-aa073b4f361f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:06:18.851540 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.851496    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:06:18.851540 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.851518    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:06:18.853258 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.853234    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-dshm" (OuterVolumeSpecName: "dshm") pod "386ef6c5-792a-44be-a8fa-aa073b4f361f" (UID: "386ef6c5-792a-44be-a8fa-aa073b4f361f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:06:18.853365 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.853288    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386ef6c5-792a-44be-a8fa-aa073b4f361f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "386ef6c5-792a-44be-a8fa-aa073b4f361f" (UID: "386ef6c5-792a-44be-a8fa-aa073b4f361f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:06:18.853365 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.853322    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386ef6c5-792a-44be-a8fa-aa073b4f361f-kube-api-access-nrvff" (OuterVolumeSpecName: "kube-api-access-nrvff") pod "386ef6c5-792a-44be-a8fa-aa073b4f361f" (UID: "386ef6c5-792a-44be-a8fa-aa073b4f361f"). InnerVolumeSpecName "kube-api-access-nrvff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:06:18.952262 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.952227    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:06:18.952262 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.952257    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/386ef6c5-792a-44be-a8fa-aa073b4f361f-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:06:18.952262 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.952266    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/386ef6c5-792a-44be-a8fa-aa073b4f361f-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:06:18.952499 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:18.952276    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrvff\" (UniqueName: \"kubernetes.io/projected/386ef6c5-792a-44be-a8fa-aa073b4f361f-kube-api-access-nrvff\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:06:19.715417 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:19.715374    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h" event={"ID":"386ef6c5-792a-44be-a8fa-aa073b4f361f","Type":"ContainerDied","Data":"af7d04bb4423a44491f5dfe1389de2ffc6dfa9d0818f4b9a7176abb6891ca611"}
Sep 05 03:06:19.715417 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:19.715412    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"
Sep 05 03:06:19.715887 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:19.715465    2557 scope.go:117] "RemoveContainer" containerID="6fd0b0e1f80f5ed511057d5e4ab17b912887be984d144349c9ae276700dd8d3f"
Sep 05 03:06:19.738940 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:19.738908    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"]
Sep 05 03:06:19.747724 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:19.747697    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vsf2h"]
Sep 05 03:06:20.991504 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:20.991469    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="386ef6c5-792a-44be-a8fa-aa073b4f361f" path="/var/lib/kubelet/pods/386ef6c5-792a-44be-a8fa-aa073b4f361f/volumes"
Sep 05 03:06:24.756198 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:24.756155    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:06:30.801100 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:30.801067    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:06:31.269386 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:31.269343    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:06:31.362987 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:31.362952    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:06:47.734761 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:47.734717    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:06:47.838059 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:06:47.838014    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:07:06.277306 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.277270    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"]
Sep 05 03:07:06.282634 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.282603    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:07:06.286690 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:07:06.286638    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 03:07:06.286690 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:07:06.286690 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 03:07:06.286690 ip-10-0-143-128 kubenswrapper[2557]:  > containerID="4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39" cmd=["/bin/sleep","15"]
Sep 05 03:07:06.286953 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.286711    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc" message="" err=<
Sep 05 03:07:06.286953 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:07:06.286953 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 03:07:06.286953 ip-10-0-143-128 kubenswrapper[2557]:  >
Sep 05 03:07:06.286953 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:07:06.286729    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 03:07:06.286953 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:07:06.286953 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 03:07:06.286953 ip-10-0-143-128 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc" podUID="9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30" containerName="main" containerID="cri-o://4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39"
Sep 05 03:07:06.286953 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.286773    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc" podUID="9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30" containerName="main" containerID="cri-o://4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39" gracePeriod=60
Sep 05 03:07:06.384544 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.384510    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:07:06.644187 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.644152    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:07:06.670739 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.670706    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-home\") pod \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") "
Sep 05 03:07:06.670739 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.670744    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-tls-certs\") pod \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") "
Sep 05 03:07:06.670967 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.670768    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-model-cache\") pod \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") "
Sep 05 03:07:06.670967 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.670836    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-tmp-dir\") pod \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") "
Sep 05 03:07:06.670967 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.670863    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7smhf\" (UniqueName: \"kubernetes.io/projected/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-kube-api-access-7smhf\") pod \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") "
Sep 05 03:07:06.670967 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.670903    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-dshm\") pod \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\" (UID: \"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30\") "
Sep 05 03:07:06.671177 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.670975    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-home" (OuterVolumeSpecName: "home") pod "9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30" (UID: "9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:07:06.671177 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.671123    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-model-cache" (OuterVolumeSpecName: "model-cache") pod "9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30" (UID: "9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:07:06.671326 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.671296    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:07:06.671399 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.671333    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:07:06.671399 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.671331    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30" (UID: "9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:07:06.673430 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.673401    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-kube-api-access-7smhf" (OuterVolumeSpecName: "kube-api-access-7smhf") pod "9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30" (UID: "9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30"). InnerVolumeSpecName "kube-api-access-7smhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:07:06.673571 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.673538    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-dshm" (OuterVolumeSpecName: "dshm") pod "9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30" (UID: "9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:07:06.673644 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.673625    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30" (UID: "9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:07:06.771741 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.771695    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:07:06.771741 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.771733    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:07:06.771741 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.771747    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7smhf\" (UniqueName: \"kubernetes.io/projected/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-kube-api-access-7smhf\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:07:06.771990 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.771760    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:07:06.961915 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.961872    2557 generic.go:356] "Generic (PLEG): container finished" podID="9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30" containerID="4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39" exitCode=0
Sep 05 03:07:06.962095 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.961922    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"
Sep 05 03:07:06.962095 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.961963    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc" event={"ID":"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30","Type":"ContainerDied","Data":"4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39"}
Sep 05 03:07:06.962095 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.962022    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc" event={"ID":"9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30","Type":"ContainerDied","Data":"fe64c820e885ebc0c52c617e2efd93c642189c407c5cf05a7fbd89ba8ff418d7"}
Sep 05 03:07:06.962095 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.962053    2557 scope.go:117] "RemoveContainer" containerID="4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39"
Sep 05 03:07:06.980247 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.980217    2557 scope.go:117] "RemoveContainer" containerID="4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39"
Sep 05 03:07:06.980666 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:07:06.980643    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39\": container with ID starting with 4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39 not found: ID does not exist" containerID="4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39"
Sep 05 03:07:06.980723 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.980676    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39"} err="failed to get container status \"4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39\": rpc error: code = NotFound desc = could not find container \"4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39\": container with ID starting with 4cf2df1aa718d5f5bde20fb971b99cc29640c4c801d0585a57eb6415b7cc5f39 not found: ID does not exist"
Sep 05 03:07:06.994281 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:06.994242    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"]
Sep 05 03:07:07.003001 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:07.002970    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qgwcc"]
Sep 05 03:07:08.990237 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:08.990199    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30" path="/var/lib/kubelet/pods/9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30/volumes"
Sep 05 03:07:32.653903 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:32.653865    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:07:56.355936 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:56.355897    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:07:56.458666 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:07:56.458632    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:08:02.835966 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:02.835927    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:08:08.758522 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:08.758482    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:08:08.858242 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:08.858207    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:08:15.106729 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.106693    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"]
Sep 05 03:08:15.107119 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.107087    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerName="istio-proxy"
Sep 05 03:08:15.107119 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.107103    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerName="istio-proxy"
Sep 05 03:08:15.107119 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.107120    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerName="istio-proxy"
Sep 05 03:08:15.107261 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.107126    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerName="istio-proxy"
Sep 05 03:08:15.107261 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.107137    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30" containerName="main"
Sep 05 03:08:15.107261 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.107148    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30" containerName="main"
Sep 05 03:08:15.107261 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.107178    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386ef6c5-792a-44be-a8fa-aa073b4f361f" containerName="main"
Sep 05 03:08:15.107261 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.107187    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="386ef6c5-792a-44be-a8fa-aa073b4f361f" containerName="main"
Sep 05 03:08:15.107261 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.107255    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerName="istio-proxy"
Sep 05 03:08:15.107455 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.107265    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bb55ab6-97a3-416b-b1d5-7f8f3c0c5e30" containerName="main"
Sep 05 03:08:15.107455 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.107273    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="386ef6c5-792a-44be-a8fa-aa073b4f361f" containerName="main"
Sep 05 03:08:15.107455 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.107379    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cc72fc2-6137-4ab5-a137-3b8621d3fd02" containerName="istio-proxy"
Sep 05 03:08:15.110406 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.110387    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.113110 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.113081    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 05 03:08:15.113110 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.113100    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-5r8vj\""
Sep 05 03:08:15.113312 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.113081    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 05 03:08:15.113312 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.113082    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:08:15.113312 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.113100    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 05 03:08:15.113312 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.113082    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 05 03:08:15.122336 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.122301    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"]
Sep 05 03:08:15.156221 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.156188    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929a7352-a497-4e1c-85f2-da279f6bb349-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.156369 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.156241    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.156369 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.156268    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929a7352-a497-4e1c-85f2-da279f6bb349-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.156369 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.156316    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.156509 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.156392    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.156509 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.156426    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.156509 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.156472    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.156509 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.156489    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64ffn\" (UniqueName: \"kubernetes.io/projected/929a7352-a497-4e1c-85f2-da279f6bb349-kube-api-access-64ffn\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.179460 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.179408    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:08:15.257787 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.257748    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.257983 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.257792    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.257983 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.257919    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.258120 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.257983    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.258120 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.258008    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64ffn\" (UniqueName: \"kubernetes.io/projected/929a7352-a497-4e1c-85f2-da279f6bb349-kube-api-access-64ffn\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.258227 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.258121    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929a7352-a497-4e1c-85f2-da279f6bb349-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.258227 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.258166    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.258227 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.258192    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.258227 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.258197    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.258227 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.258220    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929a7352-a497-4e1c-85f2-da279f6bb349-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.258470 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.258270    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.258470 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.258361    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.258873 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.258848    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929a7352-a497-4e1c-85f2-da279f6bb349-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.260506 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.260478    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.260675 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.260659    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929a7352-a497-4e1c-85f2-da279f6bb349-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.266557 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.266531    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64ffn\" (UniqueName: \"kubernetes.io/projected/929a7352-a497-4e1c-85f2-da279f6bb349-kube-api-access-64ffn\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.407528 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.407426    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:08:15.421293 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.421260    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:15.701836 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:15.701810    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"]
Sep 05 03:08:15.703477 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:08:15.703424    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929a7352_a497_4e1c_85f2_da279f6bb349.slice/crio-686525d6c6697aaad2e187bab6262406509dcaa413d7ef0d137334cda4904d1f WatchSource:0}: Error finding container 686525d6c6697aaad2e187bab6262406509dcaa413d7ef0d137334cda4904d1f: Status 404 returned error can't find the container with id 686525d6c6697aaad2e187bab6262406509dcaa413d7ef0d137334cda4904d1f
Sep 05 03:08:16.325641 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:16.325596    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" event={"ID":"929a7352-a497-4e1c-85f2-da279f6bb349","Type":"ContainerStarted","Data":"ce4f87e34d9adf0068b97222ac1bf830ad2504ada815ff122eac9c5b2fead36f"}
Sep 05 03:08:16.326126 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:16.325647    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" event={"ID":"929a7352-a497-4e1c-85f2-da279f6bb349","Type":"ContainerStarted","Data":"686525d6c6697aaad2e187bab6262406509dcaa413d7ef0d137334cda4904d1f"}
Sep 05 03:08:17.204479 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:17.204424    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:08:17.214327 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:17.214301    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:08:19.344526 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:19.344473    2557 generic.go:356] "Generic (PLEG): container finished" podID="929a7352-a497-4e1c-85f2-da279f6bb349" containerID="ce4f87e34d9adf0068b97222ac1bf830ad2504ada815ff122eac9c5b2fead36f" exitCode=0
Sep 05 03:08:19.344911 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:19.344543    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" event={"ID":"929a7352-a497-4e1c-85f2-da279f6bb349","Type":"ContainerDied","Data":"ce4f87e34d9adf0068b97222ac1bf830ad2504ada815ff122eac9c5b2fead36f"}
Sep 05 03:08:20.353786 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:20.353747    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" event={"ID":"929a7352-a497-4e1c-85f2-da279f6bb349","Type":"ContainerStarted","Data":"3c64cc27fe63e1aa5a17d34666733a0c4b860e5df909bb69918bc69d558228f1"}
Sep 05 03:08:20.380295 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:20.380230    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" podStartSLOduration=2.329815429 podStartE2EDuration="5.380210623s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.050395194s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:08:20.377984145 +0000 UTC m=+3003.933140453" watchObservedRunningTime="2026-09-05 03:08:20.380210623 +0000 UTC m=+3003.935366932"
Sep 05 03:08:20.421861 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:20.421825    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:25.421894 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:25.421846    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:08:25.423577 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:25.423545    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 05 03:08:35.422480 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:35.422423    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 05 03:08:45.422245 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:45.422198    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 05 03:08:55.422744 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:08:55.422637    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 05 03:09:05.421993 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:09:05.421936    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 05 03:09:08.273726 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:09:08.273673    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:09:10.458796 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:09:10.458699    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:09:10.561194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:09:10.561149    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:09:15.422549 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:09:15.422499    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 05 03:09:25.422073 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:09:25.422029    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 05 03:09:35.421798 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:09:35.421738    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 05 03:09:45.431913 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:09:45.431875    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:09:45.439834 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:09:45.439803    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:09:59.291121 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:09:59.291075    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:09:59.389840 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:09:59.389802    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:10:05.545634 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:05.545589    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:10:10.945423 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:10.945387    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:10:11.016688 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:11.016653    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:10:11.601223 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:11.601186    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:10:11.700773 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:11.700737    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:10:18.520748 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.520702    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"]
Sep 05 03:10:18.525732 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.525708    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.528542 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.528510    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 05 03:10:18.528679 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.528624    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-wsnts\""
Sep 05 03:10:18.528679 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.528624    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 05 03:10:18.528799 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.528722    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 05 03:10:18.532595 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.532570    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"]
Sep 05 03:10:18.541415 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.541384    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:10:18.621898 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.621864    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.622050 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.621905    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.622050 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.621925    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.622050 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.622037    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.622157 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.622071    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3718e07b-082f-4454-a964-5691696c508a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.622157 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.622105    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.622157 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.622143    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkg9c\" (UniqueName: \"kubernetes.io/projected/3718e07b-082f-4454-a964-5691696c508a-kube-api-access-rkg9c\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.722747 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.722706    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.722747 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.722749    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3718e07b-082f-4454-a964-5691696c508a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.723045 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.723013    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.723157 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.723127    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkg9c\" (UniqueName: \"kubernetes.io/projected/3718e07b-082f-4454-a964-5691696c508a-kube-api-access-rkg9c\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.723234 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.723198    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.723322 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.723204    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.723493 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.723463    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.723493 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.723483    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.723685 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.723511    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.724113 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.724083    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.724292 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.724268    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.725680 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.725654    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3718e07b-082f-4454-a964-5691696c508a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.725761 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.725705    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.739033 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.739005    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkg9c\" (UniqueName: \"kubernetes.io/projected/3718e07b-082f-4454-a964-5691696c508a-kube-api-access-rkg9c\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:18.762094 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.762044    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:10:18.837873 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:18.837834    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:19.114705 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:19.114678    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"]
Sep 05 03:10:19.151605 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:19.151580    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 03:10:20.066919 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:20.066879    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" event={"ID":"3718e07b-082f-4454-a964-5691696c508a","Type":"ContainerStarted","Data":"f4ecaee6b623772454b3dd12fcb52fb170bd2a6968a0c72604365ba9599d2625"}
Sep 05 03:10:21.729295 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:21.729260    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"]
Sep 05 03:10:21.750640 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:21.750576    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:10:21.814115 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:21.814081    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:10:27.110347 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:27.110299    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" event={"ID":"3718e07b-082f-4454-a964-5691696c508a","Type":"ContainerStarted","Data":"1be7faa3720910e143e05ed1f131e6702af1fde43de2cf71c60a2f9925e09f1e"}
Sep 05 03:10:28.117192 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:28.117150    2557 generic.go:356] "Generic (PLEG): container finished" podID="3718e07b-082f-4454-a964-5691696c508a" containerID="1be7faa3720910e143e05ed1f131e6702af1fde43de2cf71c60a2f9925e09f1e" exitCode=0
Sep 05 03:10:28.117612 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:28.117203    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" event={"ID":"3718e07b-082f-4454-a964-5691696c508a","Type":"ContainerDied","Data":"1be7faa3720910e143e05ed1f131e6702af1fde43de2cf71c60a2f9925e09f1e"}
Sep 05 03:10:29.124863 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:29.124818    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" event={"ID":"3718e07b-082f-4454-a964-5691696c508a","Type":"ContainerStarted","Data":"be57b59efde4dd89ac994a54ef7a45ddca6fcf2cdfd57c8b5233d03136ff6119"}
Sep 05 03:10:29.125213 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:29.124902    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" event={"ID":"3718e07b-082f-4454-a964-5691696c508a","Type":"ContainerStarted","Data":"ae3e6ccfd1fc860df8c14e45197c73c2177e74f97cad2ffc887eea48041f5d7c"}
Sep 05 03:10:29.149547 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:29.149490    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podStartSLOduration=2.490057769 podStartE2EDuration="11.149467829s" totalImagesPullingTime="7.474589265s" totalInitContainerRuntime="1.184820795s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:10:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:10:29.147260403 +0000 UTC m=+3132.702416710" watchObservedRunningTime="2026-09-05 03:10:29.149467829 +0000 UTC m=+3132.704624138"
Sep 05 03:10:29.774055 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:29.774012    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:10:29.838397 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:29.838354    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:36.761454 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:36.761399    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" containerName="main" containerID="cri-o://3c64cc27fe63e1aa5a17d34666733a0c4b860e5df909bb69918bc69d558228f1" gracePeriod=45
Sep 05 03:10:37.409254 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.409225    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:10:37.481259 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.481175    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-dshm\") pod \"929a7352-a497-4e1c-85f2-da279f6bb349\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") "
Sep 05 03:10:37.481259 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.481210    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-model-cache\") pod \"929a7352-a497-4e1c-85f2-da279f6bb349\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") "
Sep 05 03:10:37.481259 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.481227    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-home\") pod \"929a7352-a497-4e1c-85f2-da279f6bb349\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") "
Sep 05 03:10:37.481562 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.481283    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64ffn\" (UniqueName: \"kubernetes.io/projected/929a7352-a497-4e1c-85f2-da279f6bb349-kube-api-access-64ffn\") pod \"929a7352-a497-4e1c-85f2-da279f6bb349\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") "
Sep 05 03:10:37.481562 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.481315    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929a7352-a497-4e1c-85f2-da279f6bb349-tls-certs\") pod \"929a7352-a497-4e1c-85f2-da279f6bb349\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") "
Sep 05 03:10:37.481562 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.481341    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929a7352-a497-4e1c-85f2-da279f6bb349-cabundle-cert\") pod \"929a7352-a497-4e1c-85f2-da279f6bb349\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") "
Sep 05 03:10:37.481562 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.481365    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-tmp-dir\") pod \"929a7352-a497-4e1c-85f2-da279f6bb349\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") "
Sep 05 03:10:37.481562 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.481399    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-kserve-provision-location\") pod \"929a7352-a497-4e1c-85f2-da279f6bb349\" (UID: \"929a7352-a497-4e1c-85f2-da279f6bb349\") "
Sep 05 03:10:37.481562 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.481467    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-model-cache" (OuterVolumeSpecName: "model-cache") pod "929a7352-a497-4e1c-85f2-da279f6bb349" (UID: "929a7352-a497-4e1c-85f2-da279f6bb349"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:10:37.481869 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.481659    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-home" (OuterVolumeSpecName: "home") pod "929a7352-a497-4e1c-85f2-da279f6bb349" (UID: "929a7352-a497-4e1c-85f2-da279f6bb349"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:10:37.481869 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.481745    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:10:37.481869 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.481745    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929a7352-a497-4e1c-85f2-da279f6bb349-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "929a7352-a497-4e1c-85f2-da279f6bb349" (UID: "929a7352-a497-4e1c-85f2-da279f6bb349"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:10:37.481869 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.481762    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:10:37.483819 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.483792    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929a7352-a497-4e1c-85f2-da279f6bb349-kube-api-access-64ffn" (OuterVolumeSpecName: "kube-api-access-64ffn") pod "929a7352-a497-4e1c-85f2-da279f6bb349" (UID: "929a7352-a497-4e1c-85f2-da279f6bb349"). InnerVolumeSpecName "kube-api-access-64ffn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:10:37.483927 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.483791    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-dshm" (OuterVolumeSpecName: "dshm") pod "929a7352-a497-4e1c-85f2-da279f6bb349" (UID: "929a7352-a497-4e1c-85f2-da279f6bb349"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:10:37.483927 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.483844    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929a7352-a497-4e1c-85f2-da279f6bb349-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "929a7352-a497-4e1c-85f2-da279f6bb349" (UID: "929a7352-a497-4e1c-85f2-da279f6bb349"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:10:37.497756 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.497697    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "929a7352-a497-4e1c-85f2-da279f6bb349" (UID: "929a7352-a497-4e1c-85f2-da279f6bb349"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:10:37.505788 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.505765    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "929a7352-a497-4e1c-85f2-da279f6bb349" (UID: "929a7352-a497-4e1c-85f2-da279f6bb349"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:10:37.583001 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.582972    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929a7352-a497-4e1c-85f2-da279f6bb349-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:10:37.583001 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.582999    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:10:37.583177 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.583008    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:10:37.583177 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.583018    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929a7352-a497-4e1c-85f2-da279f6bb349-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:10:37.583177 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.583027    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64ffn\" (UniqueName: \"kubernetes.io/projected/929a7352-a497-4e1c-85f2-da279f6bb349-kube-api-access-64ffn\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:10:37.583177 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:37.583037    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929a7352-a497-4e1c-85f2-da279f6bb349-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:10:38.176801 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.176763    2557 generic.go:356] "Generic (PLEG): container finished" podID="929a7352-a497-4e1c-85f2-da279f6bb349" containerID="3c64cc27fe63e1aa5a17d34666733a0c4b860e5df909bb69918bc69d558228f1" exitCode=0
Sep 05 03:10:38.177207 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.176833    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"
Sep 05 03:10:38.177207 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.176842    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" event={"ID":"929a7352-a497-4e1c-85f2-da279f6bb349","Type":"ContainerDied","Data":"3c64cc27fe63e1aa5a17d34666733a0c4b860e5df909bb69918bc69d558228f1"}
Sep 05 03:10:38.177207 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.176897    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch" event={"ID":"929a7352-a497-4e1c-85f2-da279f6bb349","Type":"ContainerDied","Data":"686525d6c6697aaad2e187bab6262406509dcaa413d7ef0d137334cda4904d1f"}
Sep 05 03:10:38.177207 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.176921    2557 scope.go:117] "RemoveContainer" containerID="3c64cc27fe63e1aa5a17d34666733a0c4b860e5df909bb69918bc69d558228f1"
Sep 05 03:10:38.195080 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.195058    2557 scope.go:117] "RemoveContainer" containerID="ce4f87e34d9adf0068b97222ac1bf830ad2504ada815ff122eac9c5b2fead36f"
Sep 05 03:10:38.203424 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.203399    2557 scope.go:117] "RemoveContainer" containerID="3c64cc27fe63e1aa5a17d34666733a0c4b860e5df909bb69918bc69d558228f1"
Sep 05 03:10:38.203763 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:10:38.203737    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c64cc27fe63e1aa5a17d34666733a0c4b860e5df909bb69918bc69d558228f1\": container with ID starting with 3c64cc27fe63e1aa5a17d34666733a0c4b860e5df909bb69918bc69d558228f1 not found: ID does not exist" containerID="3c64cc27fe63e1aa5a17d34666733a0c4b860e5df909bb69918bc69d558228f1"
Sep 05 03:10:38.203856 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.203769    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c64cc27fe63e1aa5a17d34666733a0c4b860e5df909bb69918bc69d558228f1"} err="failed to get container status \"3c64cc27fe63e1aa5a17d34666733a0c4b860e5df909bb69918bc69d558228f1\": rpc error: code = NotFound desc = could not find container \"3c64cc27fe63e1aa5a17d34666733a0c4b860e5df909bb69918bc69d558228f1\": container with ID starting with 3c64cc27fe63e1aa5a17d34666733a0c4b860e5df909bb69918bc69d558228f1 not found: ID does not exist"
Sep 05 03:10:38.203856 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.203786    2557 scope.go:117] "RemoveContainer" containerID="ce4f87e34d9adf0068b97222ac1bf830ad2504ada815ff122eac9c5b2fead36f"
Sep 05 03:10:38.204048 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:10:38.204025    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4f87e34d9adf0068b97222ac1bf830ad2504ada815ff122eac9c5b2fead36f\": container with ID starting with ce4f87e34d9adf0068b97222ac1bf830ad2504ada815ff122eac9c5b2fead36f not found: ID does not exist" containerID="ce4f87e34d9adf0068b97222ac1bf830ad2504ada815ff122eac9c5b2fead36f"
Sep 05 03:10:38.204121 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.204053    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4f87e34d9adf0068b97222ac1bf830ad2504ada815ff122eac9c5b2fead36f"} err="failed to get container status \"ce4f87e34d9adf0068b97222ac1bf830ad2504ada815ff122eac9c5b2fead36f\": rpc error: code = NotFound desc = could not find container \"ce4f87e34d9adf0068b97222ac1bf830ad2504ada815ff122eac9c5b2fead36f\": container with ID starting with ce4f87e34d9adf0068b97222ac1bf830ad2504ada815ff122eac9c5b2fead36f not found: ID does not exist"
Sep 05 03:10:38.207875 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.207850    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"]
Sep 05 03:10:38.215713 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.215688    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-64569f67bb-rvmch"]
Sep 05 03:10:38.838399 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.838364    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:10:38.839844 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.839810    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 05 03:10:38.990533 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:38.990456    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" path="/var/lib/kubelet/pods/929a7352-a497-4e1c-85f2-da279f6bb349/volumes"
Sep 05 03:10:48.839133 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:48.839085    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 05 03:10:58.838796 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:10:58.838751    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 05 03:11:08.838946 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:08.838894    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 05 03:11:10.480948 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:10.480902    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:11:12.421617 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:12.421571    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:11:17.559796 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:17.559758    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:11:18.838982 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:18.838935    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 05 03:11:23.089168 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:23.089128    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:11:23.222812 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:23.222764    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:11:27.820681 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.820641    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"]
Sep 05 03:11:27.821276 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.821038    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" containerName="main"
Sep 05 03:11:27.821276 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.821051    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" containerName="main"
Sep 05 03:11:27.821276 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.821060    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" containerName="storage-initializer"
Sep 05 03:11:27.821276 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.821069    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" containerName="storage-initializer"
Sep 05 03:11:27.821276 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.821135    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="929a7352-a497-4e1c-85f2-da279f6bb349" containerName="main"
Sep 05 03:11:27.824420 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.824393    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.827025 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.826999    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 05 03:11:27.827353 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.827330    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 05 03:11:27.827471 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.827378    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 05 03:11:27.827875 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.827850    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:11:27.827982 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.827860    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-pbf66\""
Sep 05 03:11:27.828221 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.828205    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 05 03:11:27.837644 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.837619    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"]
Sep 05 03:11:27.838743 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.838720    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.838857 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.838763    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.838857 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.838802    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.838857 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.838840    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.838993 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.838923    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.838993 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.838969    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.839074 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.839003    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.839074 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.839042    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvj6m\" (UniqueName: \"kubernetes.io/projected/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-kube-api-access-cvj6m\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.850099 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.850037    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:11:27.939754 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.939714    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.939754 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.939762    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.940025 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.939796    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.940025 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.939846    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.940025 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.939876    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.940025 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.939909    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.940025 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.939934    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvj6m\" (UniqueName: \"kubernetes.io/projected/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-kube-api-access-cvj6m\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.940025 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.940001    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.940346 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.940203    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.940346 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.940247    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.940425 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.940393    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.940623 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.940600    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.940880 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.940844    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.942561 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.942541    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.943105 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.943076    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:27.949370 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:27.949343    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvj6m\" (UniqueName: \"kubernetes.io/projected/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-kube-api-access-cvj6m\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:28.055265 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:28.055207    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:11:28.135696 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:28.135585    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:28.424922 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:28.424891    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"]
Sep 05 03:11:28.427125 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:11:28.427086    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e14ff2c_cca4_4a72_bb0d_ea2af4005678.slice/crio-9d1a90f2aaf42bf31ff51754ce9d5a332d441f118225ebcbb98a4e078649baa0 WatchSource:0}: Error finding container 9d1a90f2aaf42bf31ff51754ce9d5a332d441f118225ebcbb98a4e078649baa0: Status 404 returned error can't find the container with id 9d1a90f2aaf42bf31ff51754ce9d5a332d441f118225ebcbb98a4e078649baa0
Sep 05 03:11:28.465360 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:28.465312    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" event={"ID":"4e14ff2c-cca4-4a72-bb0d-ea2af4005678","Type":"ContainerStarted","Data":"9d1a90f2aaf42bf31ff51754ce9d5a332d441f118225ebcbb98a4e078649baa0"}
Sep 05 03:11:28.839318 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:28.838885    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 05 03:11:29.049501 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:29.049459    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:11:29.472934 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:29.472895    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" event={"ID":"4e14ff2c-cca4-4a72-bb0d-ea2af4005678","Type":"ContainerStarted","Data":"6e182df102a7a0aebd15d10a20044f1332477ff7e0c7bc489bcacc946a2048d6"}
Sep 05 03:11:32.493601 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:32.493564    2557 generic.go:356] "Generic (PLEG): container finished" podID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerID="6e182df102a7a0aebd15d10a20044f1332477ff7e0c7bc489bcacc946a2048d6" exitCode=0
Sep 05 03:11:32.494056 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:32.493640    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" event={"ID":"4e14ff2c-cca4-4a72-bb0d-ea2af4005678","Type":"ContainerDied","Data":"6e182df102a7a0aebd15d10a20044f1332477ff7e0c7bc489bcacc946a2048d6"}
Sep 05 03:11:33.511506 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:33.511393    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" event={"ID":"4e14ff2c-cca4-4a72-bb0d-ea2af4005678","Type":"ContainerStarted","Data":"e829a145b2467a7add2a63ab5a72322aba1b6b10adc7795056b0bd9e9ad1e76a"}
Sep 05 03:11:33.540541 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:33.540472    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" podStartSLOduration=3.47277628 podStartE2EDuration="6.540450809s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.067674529s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:11:33.537455142 +0000 UTC m=+3197.092611443" watchObservedRunningTime="2026-09-05 03:11:33.540450809 +0000 UTC m=+3197.095607112"
Sep 05 03:11:34.136614 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:34.136567    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:38.136374 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:38.136328    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:11:38.138191 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:38.138159    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 05 03:11:38.838868 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:38.838807    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 05 03:11:46.253479 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:46.253414    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:11:48.137039 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:48.136994    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 05 03:11:48.838482 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:48.838403    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 05 03:11:58.136809 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:58.136760    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 05 03:11:58.838636 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:11:58.838588    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 05 03:12:08.136449 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:08.136369    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 05 03:12:08.839403 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:08.839350    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 05 03:12:18.136448 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:18.136401    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 05 03:12:18.839476 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:18.839395    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 05 03:12:28.136591 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:28.136538    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 05 03:12:28.850412 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:28.850374    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:12:28.859123 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:28.859090    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:12:34.739164 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:34.739111    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"]
Sep 05 03:12:34.739837 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:34.739779    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="modelcar" containerID="cri-o://be57b59efde4dd89ac994a54ef7a45ddca6fcf2cdfd57c8b5233d03136ff6119" gracePeriod=60
Sep 05 03:12:34.745276 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:34.743642    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:12:34.836603 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:34.836565    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:12:34.938688 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:34.938651    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw_3718e07b-082f-4454-a964-5691696c508a/modelcar/0.log"
Sep 05 03:12:34.939697 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:34.939574    2557 generic.go:356] "Generic (PLEG): container finished" podID="3718e07b-082f-4454-a964-5691696c508a" containerID="be57b59efde4dd89ac994a54ef7a45ddca6fcf2cdfd57c8b5233d03136ff6119" exitCode=143
Sep 05 03:12:34.939697 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:34.939614    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" event={"ID":"3718e07b-082f-4454-a964-5691696c508a","Type":"ContainerDied","Data":"be57b59efde4dd89ac994a54ef7a45ddca6fcf2cdfd57c8b5233d03136ff6119"}
Sep 05 03:12:38.136638 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:38.136586    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 05 03:12:44.274364 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:44.274319    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:12:48.136575 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:48.136518    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 05 03:12:49.767394 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:49.767325    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main" containerID="cri-o://ae3e6ccfd1fc860df8c14e45197c73c2177e74f97cad2ffc887eea48041f5d7c" gracePeriod=45
Sep 05 03:12:49.851645 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:49.851584    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 05 03:12:50.780414 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.780385    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw_3718e07b-082f-4454-a964-5691696c508a/modelcar/0.log"
Sep 05 03:12:50.783725 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.783695    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:12:50.860404 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.860369    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-kserve-provision-location\") pod \"3718e07b-082f-4454-a964-5691696c508a\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") "
Sep 05 03:12:50.860614 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.860464    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-model-cache\") pod \"3718e07b-082f-4454-a964-5691696c508a\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") "
Sep 05 03:12:50.860614 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.860484    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3718e07b-082f-4454-a964-5691696c508a-tls-certs\") pod \"3718e07b-082f-4454-a964-5691696c508a\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") "
Sep 05 03:12:50.860614 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.860520    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-tmp-dir\") pod \"3718e07b-082f-4454-a964-5691696c508a\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") "
Sep 05 03:12:50.860614 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.860545    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-home\") pod \"3718e07b-082f-4454-a964-5691696c508a\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") "
Sep 05 03:12:50.860614 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.860580    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkg9c\" (UniqueName: \"kubernetes.io/projected/3718e07b-082f-4454-a964-5691696c508a-kube-api-access-rkg9c\") pod \"3718e07b-082f-4454-a964-5691696c508a\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") "
Sep 05 03:12:50.860614 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.860616    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-dshm\") pod \"3718e07b-082f-4454-a964-5691696c508a\" (UID: \"3718e07b-082f-4454-a964-5691696c508a\") "
Sep 05 03:12:50.860913 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.860777    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3718e07b-082f-4454-a964-5691696c508a" (UID: "3718e07b-082f-4454-a964-5691696c508a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:12:50.860913 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.860788    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-model-cache" (OuterVolumeSpecName: "model-cache") pod "3718e07b-082f-4454-a964-5691696c508a" (UID: "3718e07b-082f-4454-a964-5691696c508a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:12:50.861011 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.860917    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:12:50.861011 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.860940    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:12:50.861619 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.861583    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-home" (OuterVolumeSpecName: "home") pod "3718e07b-082f-4454-a964-5691696c508a" (UID: "3718e07b-082f-4454-a964-5691696c508a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:12:50.863303 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.863263    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3718e07b-082f-4454-a964-5691696c508a-kube-api-access-rkg9c" (OuterVolumeSpecName: "kube-api-access-rkg9c") pod "3718e07b-082f-4454-a964-5691696c508a" (UID: "3718e07b-082f-4454-a964-5691696c508a"). InnerVolumeSpecName "kube-api-access-rkg9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:12:50.863449 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.863347    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3718e07b-082f-4454-a964-5691696c508a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3718e07b-082f-4454-a964-5691696c508a" (UID: "3718e07b-082f-4454-a964-5691696c508a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:12:50.863524 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.863493    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-dshm" (OuterVolumeSpecName: "dshm") pod "3718e07b-082f-4454-a964-5691696c508a" (UID: "3718e07b-082f-4454-a964-5691696c508a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:12:50.875354 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.875316    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3718e07b-082f-4454-a964-5691696c508a" (UID: "3718e07b-082f-4454-a964-5691696c508a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:12:50.962389 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.962350    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkg9c\" (UniqueName: \"kubernetes.io/projected/3718e07b-082f-4454-a964-5691696c508a-kube-api-access-rkg9c\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:12:50.962389 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.962385    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:12:50.962607 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.962398    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3718e07b-082f-4454-a964-5691696c508a-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:12:50.962607 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.962413    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:12:50.962607 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:50.962421    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3718e07b-082f-4454-a964-5691696c508a-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:12:51.036423 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.036338    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw_3718e07b-082f-4454-a964-5691696c508a/modelcar/0.log"
Sep 05 03:12:51.037129 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.037100    2557 generic.go:356] "Generic (PLEG): container finished" podID="3718e07b-082f-4454-a964-5691696c508a" containerID="ae3e6ccfd1fc860df8c14e45197c73c2177e74f97cad2ffc887eea48041f5d7c" exitCode=0
Sep 05 03:12:51.037252 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.037175    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"
Sep 05 03:12:51.037252 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.037187    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" event={"ID":"3718e07b-082f-4454-a964-5691696c508a","Type":"ContainerDied","Data":"ae3e6ccfd1fc860df8c14e45197c73c2177e74f97cad2ffc887eea48041f5d7c"}
Sep 05 03:12:51.037341 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.037255    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw" event={"ID":"3718e07b-082f-4454-a964-5691696c508a","Type":"ContainerDied","Data":"f4ecaee6b623772454b3dd12fcb52fb170bd2a6968a0c72604365ba9599d2625"}
Sep 05 03:12:51.037341 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.037283    2557 scope.go:117] "RemoveContainer" containerID="be57b59efde4dd89ac994a54ef7a45ddca6fcf2cdfd57c8b5233d03136ff6119"
Sep 05 03:12:51.048647 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.048626    2557 scope.go:117] "RemoveContainer" containerID="ae3e6ccfd1fc860df8c14e45197c73c2177e74f97cad2ffc887eea48041f5d7c"
Sep 05 03:12:51.056563 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.056526    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"]
Sep 05 03:12:51.058013 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.057989    2557 scope.go:117] "RemoveContainer" containerID="1be7faa3720910e143e05ed1f131e6702af1fde43de2cf71c60a2f9925e09f1e"
Sep 05 03:12:51.059975 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.059953    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fzkrmw"]
Sep 05 03:12:51.066099 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.066083    2557 scope.go:117] "RemoveContainer" containerID="be57b59efde4dd89ac994a54ef7a45ddca6fcf2cdfd57c8b5233d03136ff6119"
Sep 05 03:12:51.066403 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:12:51.066383    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be57b59efde4dd89ac994a54ef7a45ddca6fcf2cdfd57c8b5233d03136ff6119\": container with ID starting with be57b59efde4dd89ac994a54ef7a45ddca6fcf2cdfd57c8b5233d03136ff6119 not found: ID does not exist" containerID="be57b59efde4dd89ac994a54ef7a45ddca6fcf2cdfd57c8b5233d03136ff6119"
Sep 05 03:12:51.066504 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.066422    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be57b59efde4dd89ac994a54ef7a45ddca6fcf2cdfd57c8b5233d03136ff6119"} err="failed to get container status \"be57b59efde4dd89ac994a54ef7a45ddca6fcf2cdfd57c8b5233d03136ff6119\": rpc error: code = NotFound desc = could not find container \"be57b59efde4dd89ac994a54ef7a45ddca6fcf2cdfd57c8b5233d03136ff6119\": container with ID starting with be57b59efde4dd89ac994a54ef7a45ddca6fcf2cdfd57c8b5233d03136ff6119 not found: ID does not exist"
Sep 05 03:12:51.066504 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.066460    2557 scope.go:117] "RemoveContainer" containerID="ae3e6ccfd1fc860df8c14e45197c73c2177e74f97cad2ffc887eea48041f5d7c"
Sep 05 03:12:51.066768 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:12:51.066740    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae3e6ccfd1fc860df8c14e45197c73c2177e74f97cad2ffc887eea48041f5d7c\": container with ID starting with ae3e6ccfd1fc860df8c14e45197c73c2177e74f97cad2ffc887eea48041f5d7c not found: ID does not exist" containerID="ae3e6ccfd1fc860df8c14e45197c73c2177e74f97cad2ffc887eea48041f5d7c"
Sep 05 03:12:51.066818 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.066773    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3e6ccfd1fc860df8c14e45197c73c2177e74f97cad2ffc887eea48041f5d7c"} err="failed to get container status \"ae3e6ccfd1fc860df8c14e45197c73c2177e74f97cad2ffc887eea48041f5d7c\": rpc error: code = NotFound desc = could not find container \"ae3e6ccfd1fc860df8c14e45197c73c2177e74f97cad2ffc887eea48041f5d7c\": container with ID starting with ae3e6ccfd1fc860df8c14e45197c73c2177e74f97cad2ffc887eea48041f5d7c not found: ID does not exist"
Sep 05 03:12:51.066818 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.066789    2557 scope.go:117] "RemoveContainer" containerID="1be7faa3720910e143e05ed1f131e6702af1fde43de2cf71c60a2f9925e09f1e"
Sep 05 03:12:51.067094 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:12:51.067074    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be7faa3720910e143e05ed1f131e6702af1fde43de2cf71c60a2f9925e09f1e\": container with ID starting with 1be7faa3720910e143e05ed1f131e6702af1fde43de2cf71c60a2f9925e09f1e not found: ID does not exist" containerID="1be7faa3720910e143e05ed1f131e6702af1fde43de2cf71c60a2f9925e09f1e"
Sep 05 03:12:51.067147 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:51.067102    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be7faa3720910e143e05ed1f131e6702af1fde43de2cf71c60a2f9925e09f1e"} err="failed to get container status \"1be7faa3720910e143e05ed1f131e6702af1fde43de2cf71c60a2f9925e09f1e\": rpc error: code = NotFound desc = could not find container \"1be7faa3720910e143e05ed1f131e6702af1fde43de2cf71c60a2f9925e09f1e\": container with ID starting with 1be7faa3720910e143e05ed1f131e6702af1fde43de2cf71c60a2f9925e09f1e not found: ID does not exist"
Sep 05 03:12:52.063555 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:52.063515    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:12:52.997683 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:52.997639    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3718e07b-082f-4454-a964-5691696c508a" path="/var/lib/kubelet/pods/3718e07b-082f-4454-a964-5691696c508a/volumes"
Sep 05 03:12:57.623521 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:57.623477    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:12:57.678777 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:57.678733    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:12:58.146505 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:58.146471    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:12:58.154804 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:58.154780    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:12:59.577667 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:59.577627    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:12:59.626904 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:59.626859    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"]
Sep 05 03:12:59.682869 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:12:59.682832    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:15.113708 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.113660    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerName="main" containerID="cri-o://e829a145b2467a7add2a63ab5a72322aba1b6b10adc7795056b0bd9e9ad1e76a" gracePeriod=45
Sep 05 03:13:15.773452 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.773373    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:13:15.889957 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.889914    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-dshm\") pod \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") "
Sep 05 03:13:15.890159 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.890028    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-model-cache\") pod \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") "
Sep 05 03:13:15.890159 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.890059    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-tls-certs\") pod \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") "
Sep 05 03:13:15.890159 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.890083    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvj6m\" (UniqueName: \"kubernetes.io/projected/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-kube-api-access-cvj6m\") pod \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") "
Sep 05 03:13:15.890159 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.890116    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-kserve-provision-location\") pod \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") "
Sep 05 03:13:15.890159 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.890142    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-tmp-dir\") pod \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") "
Sep 05 03:13:15.890429 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.890166    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-cabundle-cert\") pod \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") "
Sep 05 03:13:15.890429 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.890245    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-home\") pod \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\" (UID: \"4e14ff2c-cca4-4a72-bb0d-ea2af4005678\") "
Sep 05 03:13:15.890429 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.890349    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e14ff2c-cca4-4a72-bb0d-ea2af4005678" (UID: "4e14ff2c-cca4-4a72-bb0d-ea2af4005678"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:13:15.890636 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.890566    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:13:15.890773 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.890742    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e14ff2c-cca4-4a72-bb0d-ea2af4005678" (UID: "4e14ff2c-cca4-4a72-bb0d-ea2af4005678"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:13:15.890840 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.890772    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-home" (OuterVolumeSpecName: "home") pod "4e14ff2c-cca4-4a72-bb0d-ea2af4005678" (UID: "4e14ff2c-cca4-4a72-bb0d-ea2af4005678"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:13:15.890921 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.890890    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e14ff2c-cca4-4a72-bb0d-ea2af4005678" (UID: "4e14ff2c-cca4-4a72-bb0d-ea2af4005678"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:13:15.892735 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.892705    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-dshm" (OuterVolumeSpecName: "dshm") pod "4e14ff2c-cca4-4a72-bb0d-ea2af4005678" (UID: "4e14ff2c-cca4-4a72-bb0d-ea2af4005678"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:13:15.892853 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.892775    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e14ff2c-cca4-4a72-bb0d-ea2af4005678" (UID: "4e14ff2c-cca4-4a72-bb0d-ea2af4005678"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:13:15.892853 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.892807    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-kube-api-access-cvj6m" (OuterVolumeSpecName: "kube-api-access-cvj6m") pod "4e14ff2c-cca4-4a72-bb0d-ea2af4005678" (UID: "4e14ff2c-cca4-4a72-bb0d-ea2af4005678"). InnerVolumeSpecName "kube-api-access-cvj6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:13:15.905580 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.905551    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e14ff2c-cca4-4a72-bb0d-ea2af4005678" (UID: "4e14ff2c-cca4-4a72-bb0d-ea2af4005678"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:13:15.991757 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.991717    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:13:15.991757 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.991750    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvj6m\" (UniqueName: \"kubernetes.io/projected/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-kube-api-access-cvj6m\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:13:15.991757 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.991762    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:13:15.991985 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.991792    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:13:15.991985 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.991802    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:13:15.991985 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.991811    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:13:15.991985 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:15.991819    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e14ff2c-cca4-4a72-bb0d-ea2af4005678-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:13:16.176985 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:16.176941    2557 generic.go:356] "Generic (PLEG): container finished" podID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerID="e829a145b2467a7add2a63ab5a72322aba1b6b10adc7795056b0bd9e9ad1e76a" exitCode=0
Sep 05 03:13:16.177383 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:16.176988    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"
Sep 05 03:13:16.177383 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:16.177023    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" event={"ID":"4e14ff2c-cca4-4a72-bb0d-ea2af4005678","Type":"ContainerDied","Data":"e829a145b2467a7add2a63ab5a72322aba1b6b10adc7795056b0bd9e9ad1e76a"}
Sep 05 03:13:16.177383 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:16.177086    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg" event={"ID":"4e14ff2c-cca4-4a72-bb0d-ea2af4005678","Type":"ContainerDied","Data":"9d1a90f2aaf42bf31ff51754ce9d5a332d441f118225ebcbb98a4e078649baa0"}
Sep 05 03:13:16.177383 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:16.177115    2557 scope.go:117] "RemoveContainer" containerID="e829a145b2467a7add2a63ab5a72322aba1b6b10adc7795056b0bd9e9ad1e76a"
Sep 05 03:13:16.186942 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:16.186920    2557 scope.go:117] "RemoveContainer" containerID="6e182df102a7a0aebd15d10a20044f1332477ff7e0c7bc489bcacc946a2048d6"
Sep 05 03:13:16.196200 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:16.196179    2557 scope.go:117] "RemoveContainer" containerID="e829a145b2467a7add2a63ab5a72322aba1b6b10adc7795056b0bd9e9ad1e76a"
Sep 05 03:13:16.196531 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:13:16.196510    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e829a145b2467a7add2a63ab5a72322aba1b6b10adc7795056b0bd9e9ad1e76a\": container with ID starting with e829a145b2467a7add2a63ab5a72322aba1b6b10adc7795056b0bd9e9ad1e76a not found: ID does not exist" containerID="e829a145b2467a7add2a63ab5a72322aba1b6b10adc7795056b0bd9e9ad1e76a"
Sep 05 03:13:16.196592 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:16.196540    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e829a145b2467a7add2a63ab5a72322aba1b6b10adc7795056b0bd9e9ad1e76a"} err="failed to get container status \"e829a145b2467a7add2a63ab5a72322aba1b6b10adc7795056b0bd9e9ad1e76a\": rpc error: code = NotFound desc = could not find container \"e829a145b2467a7add2a63ab5a72322aba1b6b10adc7795056b0bd9e9ad1e76a\": container with ID starting with e829a145b2467a7add2a63ab5a72322aba1b6b10adc7795056b0bd9e9ad1e76a not found: ID does not exist"
Sep 05 03:13:16.196592 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:16.196560    2557 scope.go:117] "RemoveContainer" containerID="6e182df102a7a0aebd15d10a20044f1332477ff7e0c7bc489bcacc946a2048d6"
Sep 05 03:13:16.196800 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:13:16.196786    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e182df102a7a0aebd15d10a20044f1332477ff7e0c7bc489bcacc946a2048d6\": container with ID starting with 6e182df102a7a0aebd15d10a20044f1332477ff7e0c7bc489bcacc946a2048d6 not found: ID does not exist" containerID="6e182df102a7a0aebd15d10a20044f1332477ff7e0c7bc489bcacc946a2048d6"
Sep 05 03:13:16.196852 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:16.196803    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e182df102a7a0aebd15d10a20044f1332477ff7e0c7bc489bcacc946a2048d6"} err="failed to get container status \"6e182df102a7a0aebd15d10a20044f1332477ff7e0c7bc489bcacc946a2048d6\": rpc error: code = NotFound desc = could not find container \"6e182df102a7a0aebd15d10a20044f1332477ff7e0c7bc489bcacc946a2048d6\": container with ID starting with 6e182df102a7a0aebd15d10a20044f1332477ff7e0c7bc489bcacc946a2048d6 not found: ID does not exist"
Sep 05 03:13:16.199995 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:16.199968    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"]
Sep 05 03:13:16.203483 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:16.203460    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55bfb445d5-gf8xg"]
Sep 05 03:13:16.990710 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:16.990676    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" path="/var/lib/kubelet/pods/4e14ff2c-cca4-4a72-bb0d-ea2af4005678/volumes"
Sep 05 03:13:17.240684 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:17.240653    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:13:17.252374 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:17.252298    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:13:24.718409 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:24.718367    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:24.817095 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:24.817035    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:30.753172 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:30.753133    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:32.910925 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.910883    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"]
Sep 05 03:13:32.911291 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.911257    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerName="storage-initializer"
Sep 05 03:13:32.911291 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.911271    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerName="storage-initializer"
Sep 05 03:13:32.911291 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.911283    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerName="main"
Sep 05 03:13:32.911291 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.911290    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerName="main"
Sep 05 03:13:32.911426 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.911315    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main"
Sep 05 03:13:32.911426 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.911322    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main"
Sep 05 03:13:32.911426 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.911329    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="modelcar-init"
Sep 05 03:13:32.911426 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.911335    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="modelcar-init"
Sep 05 03:13:32.911426 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.911344    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="modelcar"
Sep 05 03:13:32.911426 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.911349    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="modelcar"
Sep 05 03:13:32.911426 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.911403    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="main"
Sep 05 03:13:32.911426 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.911415    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3718e07b-082f-4454-a964-5691696c508a" containerName="modelcar"
Sep 05 03:13:32.911426 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.911422    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e14ff2c-cca4-4a72-bb0d-ea2af4005678" containerName="main"
Sep 05 03:13:32.914012 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.913991    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:32.916561 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.916539    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 05 03:13:32.916870 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.916825    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 05 03:13:32.916993 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.916865    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 05 03:13:32.916993 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.916972    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-jncd2\""
Sep 05 03:13:32.917101 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.916975    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 05 03:13:32.917146 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.917131    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:13:32.925753 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.925728    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-dshm\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:32.925856 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.925764    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-tmp-dir\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:32.925856 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.925782    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7946ec2b-ac41-4017-a743-e19ada9d7709-cabundle-cert\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:32.925856 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.925812    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-model-cache\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:32.925856 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.925837    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-home\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:32.926031 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.925882    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7946ec2b-ac41-4017-a743-e19ada9d7709-tls-certs\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:32.926031 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.925957    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:32.926031 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.925990    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxzpt\" (UniqueName: \"kubernetes.io/projected/7946ec2b-ac41-4017-a743-e19ada9d7709-kube-api-access-wxzpt\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:32.926164 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.926115    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"]
Sep 05 03:13:32.950360 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:32.950322    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:33.026566 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.026536    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-dshm\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.026739 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.026592    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-tmp-dir\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.026739 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.026617    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7946ec2b-ac41-4017-a743-e19ada9d7709-cabundle-cert\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.026739 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.026682    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-model-cache\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.026739 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.026706    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-home\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.026739 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.026731    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7946ec2b-ac41-4017-a743-e19ada9d7709-tls-certs\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.027001 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.026829    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.027001 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.026901    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxzpt\" (UniqueName: \"kubernetes.io/projected/7946ec2b-ac41-4017-a743-e19ada9d7709-kube-api-access-wxzpt\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.027182 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.027161    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.027261 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.027220    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-home\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.027490 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.027466    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-tmp-dir\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.027728 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.027707    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-model-cache\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.027958 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.027934    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7946ec2b-ac41-4017-a743-e19ada9d7709-cabundle-cert\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.029139 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.029115    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-dshm\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.029541 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.029524    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7946ec2b-ac41-4017-a743-e19ada9d7709-tls-certs\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.036667 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.036614    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxzpt\" (UniqueName: \"kubernetes.io/projected/7946ec2b-ac41-4017-a743-e19ada9d7709-kube-api-access-wxzpt\") pod \"custom-route-timeout-test-kserve-597d4f555f-nbcqt\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.209392 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.209298    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:33.225117 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.225084    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:33.506389 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:33.506359    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"]
Sep 05 03:13:33.507984 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:13:33.507946    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7946ec2b_ac41_4017_a743_e19ada9d7709.slice/crio-07411336a8d99e2228ed1cf80a9fe1d3aebe01b9fef3131e920807ca1f16c64c WatchSource:0}: Error finding container 07411336a8d99e2228ed1cf80a9fe1d3aebe01b9fef3131e920807ca1f16c64c: Status 404 returned error can't find the container with id 07411336a8d99e2228ed1cf80a9fe1d3aebe01b9fef3131e920807ca1f16c64c
Sep 05 03:13:34.276565 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:34.276516    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" event={"ID":"7946ec2b-ac41-4017-a743-e19ada9d7709","Type":"ContainerStarted","Data":"65ba8e6d335e3e0bf49bc4ed30d2ead62d3029bb560ee5deff9ff47d45ff771b"}
Sep 05 03:13:34.277027 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:34.276580    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" event={"ID":"7946ec2b-ac41-4017-a743-e19ada9d7709","Type":"ContainerStarted","Data":"07411336a8d99e2228ed1cf80a9fe1d3aebe01b9fef3131e920807ca1f16c64c"}
Sep 05 03:13:36.291510 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:36.291404    2557 generic.go:356] "Generic (PLEG): container finished" podID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerID="65ba8e6d335e3e0bf49bc4ed30d2ead62d3029bb560ee5deff9ff47d45ff771b" exitCode=0
Sep 05 03:13:36.291510 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:36.291477    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" event={"ID":"7946ec2b-ac41-4017-a743-e19ada9d7709","Type":"ContainerDied","Data":"65ba8e6d335e3e0bf49bc4ed30d2ead62d3029bb560ee5deff9ff47d45ff771b"}
Sep 05 03:13:36.542227 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:36.542139    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:36.609936 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:36.609894    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:37.301179 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:37.301130    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" event={"ID":"7946ec2b-ac41-4017-a743-e19ada9d7709","Type":"ContainerStarted","Data":"7ca00187e6628afeb267a123ca728fc2331828a4b46baf408cb4594512ec984d"}
Sep 05 03:13:37.324301 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:37.324237    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" podStartSLOduration=2.945963233 podStartE2EDuration="5.324220654s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.378257421s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:13:37.321759088 +0000 UTC m=+3320.876915398" watchObservedRunningTime="2026-09-05 03:13:37.324220654 +0000 UTC m=+3320.879376963"
Sep 05 03:13:38.226052 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:38.226007    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:43.226268 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:43.226172    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:13:43.228019 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:43.227985    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 05 03:13:43.925956 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:43.925914    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:49.661988 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:49.661941    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:49.762159 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:49.762110    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:49.871992 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:49.871952    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:51.915621 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:51.915565    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:52.016328 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:52.016286    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:52.170533 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:52.170424    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:53.226652 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:53.226610    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 05 03:13:55.683331 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:55.683279    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:57.585284 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:57.585244    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:13:57.684259 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:13:57.684222    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:01.121882 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:01.121844    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:01.186166 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:01.186131    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:02.117632 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.117593    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"]
Sep 05 03:14:02.128069 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.128035    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.130575 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.130539    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 05 03:14:02.130815 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.130787    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 05 03:14:02.130815 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.130807    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 05 03:14:02.130985 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.130808    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-zdhn6\""
Sep 05 03:14:02.131129 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.131110    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 05 03:14:02.132876 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.132853    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"]
Sep 05 03:14:02.138763 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.138730    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:02.203390 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.203347    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg57q\" (UniqueName: \"kubernetes.io/projected/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-kube-api-access-mg57q\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.203390 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.203396    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-model-cache\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.203741 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.203423    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-home\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.203741 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.203470    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-dshm\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.203741 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.203494    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-tls-certs\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.203741 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.203510    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.203741 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.203649    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.304549 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.304502    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.304759 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.304586    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg57q\" (UniqueName: \"kubernetes.io/projected/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-kube-api-access-mg57q\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.304759 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.304631    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-model-cache\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.304759 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.304662    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-home\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.304759 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.304686    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-dshm\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.304759 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.304722    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-tls-certs\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.304759 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.304744    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.305124 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.304904    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.305124 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.304977    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-model-cache\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.305124 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.305034    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.305267 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.305241    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-home\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.307030 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.307009    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-dshm\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.307494 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.307471    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-tls-certs\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.314366 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.314340    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg57q\" (UniqueName: \"kubernetes.io/projected/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-kube-api-access-mg57q\") pod \"e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.363860 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.363820    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:02.440865 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.440764    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:02.720852 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:02.720820    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"]
Sep 05 03:14:02.722763 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:14:02.722733    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725e3bfd_2cdf_4579_9a7f_f1558ad6d95c.slice/crio-376b7a2a5e2097cbafea73467ca1a9eff6d23c5754b68e191d3f36a42a51f147 WatchSource:0}: Error finding container 376b7a2a5e2097cbafea73467ca1a9eff6d23c5754b68e191d3f36a42a51f147: Status 404 returned error can't find the container with id 376b7a2a5e2097cbafea73467ca1a9eff6d23c5754b68e191d3f36a42a51f147
Sep 05 03:14:03.226610 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:03.226562    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 05 03:14:03.448041 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:03.448005    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9" event={"ID":"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c","Type":"ContainerStarted","Data":"1b8caa703a6975de5210b8b3610241940fd700f2d2c2772d70e3ad0bc28cb6ca"}
Sep 05 03:14:03.448209 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:03.448069    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9" event={"ID":"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c","Type":"ContainerStarted","Data":"376b7a2a5e2097cbafea73467ca1a9eff6d23c5754b68e191d3f36a42a51f147"}
Sep 05 03:14:05.207057 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:05.207018    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"]
Sep 05 03:14:05.212466 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:05.212420    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:05.302409 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:05.302372    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:05.460939 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:05.460829    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9" podUID="725e3bfd-2cdf-4579-9a7f-f1558ad6d95c" containerName="storage-initializer" containerID="cri-o://1b8caa703a6975de5210b8b3610241940fd700f2d2c2772d70e3ad0bc28cb6ca" gracePeriod=60
Sep 05 03:14:06.537414 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:06.537358    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:06.641094 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:06.641055    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:06.739694 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:06.739650    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:07.830194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.830154    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:07.959547 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.959430    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-tmp-dir\") pod \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") "
Sep 05 03:14:07.959547 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.959522    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-dshm\") pod \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") "
Sep 05 03:14:07.959547 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.959540    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-kserve-provision-location\") pod \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") "
Sep 05 03:14:07.959835 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.959562    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-home\") pod \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") "
Sep 05 03:14:07.959835 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.959605    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-tls-certs\") pod \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") "
Sep 05 03:14:07.959835 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.959672    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-model-cache\") pod \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") "
Sep 05 03:14:07.959835 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.959708    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg57q\" (UniqueName: \"kubernetes.io/projected/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-kube-api-access-mg57q\") pod \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\" (UID: \"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c\") "
Sep 05 03:14:07.959835 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.959713    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "725e3bfd-2cdf-4579-9a7f-f1558ad6d95c" (UID: "725e3bfd-2cdf-4579-9a7f-f1558ad6d95c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:14:07.959835 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.959828    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-home" (OuterVolumeSpecName: "home") pod "725e3bfd-2cdf-4579-9a7f-f1558ad6d95c" (UID: "725e3bfd-2cdf-4579-9a7f-f1558ad6d95c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:14:07.960114 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.960036    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:14:07.960114 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.960059    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:14:07.960197 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.960129    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-model-cache" (OuterVolumeSpecName: "model-cache") pod "725e3bfd-2cdf-4579-9a7f-f1558ad6d95c" (UID: "725e3bfd-2cdf-4579-9a7f-f1558ad6d95c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:14:07.962000 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.961975    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "725e3bfd-2cdf-4579-9a7f-f1558ad6d95c" (UID: "725e3bfd-2cdf-4579-9a7f-f1558ad6d95c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:14:07.962225 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.962195    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-dshm" (OuterVolumeSpecName: "dshm") pod "725e3bfd-2cdf-4579-9a7f-f1558ad6d95c" (UID: "725e3bfd-2cdf-4579-9a7f-f1558ad6d95c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:14:07.962348 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:07.962290    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-kube-api-access-mg57q" (OuterVolumeSpecName: "kube-api-access-mg57q") pod "725e3bfd-2cdf-4579-9a7f-f1558ad6d95c" (UID: "725e3bfd-2cdf-4579-9a7f-f1558ad6d95c"). InnerVolumeSpecName "kube-api-access-mg57q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:14:08.031127 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.031067    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "725e3bfd-2cdf-4579-9a7f-f1558ad6d95c" (UID: "725e3bfd-2cdf-4579-9a7f-f1558ad6d95c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:14:08.061264 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.061222    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:14:08.061264 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.061250    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:14:08.061264 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.061260    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg57q\" (UniqueName: \"kubernetes.io/projected/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-kube-api-access-mg57q\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:14:08.061264 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.061272    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:14:08.061655 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.061281    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:14:08.480379 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.480336    2557 generic.go:356] "Generic (PLEG): container finished" podID="725e3bfd-2cdf-4579-9a7f-f1558ad6d95c" containerID="1b8caa703a6975de5210b8b3610241940fd700f2d2c2772d70e3ad0bc28cb6ca" exitCode=0
Sep 05 03:14:08.480599 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.480384    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"
Sep 05 03:14:08.480599 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.480418    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9" event={"ID":"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c","Type":"ContainerDied","Data":"1b8caa703a6975de5210b8b3610241940fd700f2d2c2772d70e3ad0bc28cb6ca"}
Sep 05 03:14:08.480599 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.480481    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9" event={"ID":"725e3bfd-2cdf-4579-9a7f-f1558ad6d95c","Type":"ContainerDied","Data":"376b7a2a5e2097cbafea73467ca1a9eff6d23c5754b68e191d3f36a42a51f147"}
Sep 05 03:14:08.480599 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.480505    2557 scope.go:117] "RemoveContainer" containerID="1b8caa703a6975de5210b8b3610241940fd700f2d2c2772d70e3ad0bc28cb6ca"
Sep 05 03:14:08.491334 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.491311    2557 scope.go:117] "RemoveContainer" containerID="1b8caa703a6975de5210b8b3610241940fd700f2d2c2772d70e3ad0bc28cb6ca"
Sep 05 03:14:08.491671 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:14:08.491648    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b8caa703a6975de5210b8b3610241940fd700f2d2c2772d70e3ad0bc28cb6ca\": container with ID starting with 1b8caa703a6975de5210b8b3610241940fd700f2d2c2772d70e3ad0bc28cb6ca not found: ID does not exist" containerID="1b8caa703a6975de5210b8b3610241940fd700f2d2c2772d70e3ad0bc28cb6ca"
Sep 05 03:14:08.491751 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.491679    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8caa703a6975de5210b8b3610241940fd700f2d2c2772d70e3ad0bc28cb6ca"} err="failed to get container status \"1b8caa703a6975de5210b8b3610241940fd700f2d2c2772d70e3ad0bc28cb6ca\": rpc error: code = NotFound desc = could not find container \"1b8caa703a6975de5210b8b3610241940fd700f2d2c2772d70e3ad0bc28cb6ca\": container with ID starting with 1b8caa703a6975de5210b8b3610241940fd700f2d2c2772d70e3ad0bc28cb6ca not found: ID does not exist"
Sep 05 03:14:08.522549 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.522499    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"]
Sep 05 03:14:08.526113 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.526081    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-5dcf55b4bb-flsj9"]
Sep 05 03:14:08.992093 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:08.992055    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="725e3bfd-2cdf-4579-9a7f-f1558ad6d95c" path="/var/lib/kubelet/pods/725e3bfd-2cdf-4579-9a7f-f1558ad6d95c/volumes"
Sep 05 03:14:10.797975 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:10.797941    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:10.900623 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:10.900588    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:12.224822 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:12.224784    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:12.277564 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:12.277528    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:13.225709 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:13.225666    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 05 03:14:15.396451 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:15.396389    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:15.598545 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:15.598508    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:15.698238 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:15.698147    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:17.117782 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:17.117729    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:17.220196 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:17.220160    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:21.086699 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:21.086653    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:21.148568 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:21.148532    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:22.314384 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.314346    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"]
Sep 05 03:14:22.315127 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.315100    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="725e3bfd-2cdf-4579-9a7f-f1558ad6d95c" containerName="storage-initializer"
Sep 05 03:14:22.315238 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.315131    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="725e3bfd-2cdf-4579-9a7f-f1558ad6d95c" containerName="storage-initializer"
Sep 05 03:14:22.315278 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.315253    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="725e3bfd-2cdf-4579-9a7f-f1558ad6d95c" containerName="storage-initializer"
Sep 05 03:14:22.320522 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.320487    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"
Sep 05 03:14:22.322957 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.322928    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Sep 05 03:14:22.323135 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.323114    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 05 03:14:22.323298 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.323192    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 05 03:14:22.324203 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.324072    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-qgpqf\""
Sep 05 03:14:22.327027 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.327001    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"]
Sep 05 03:14:22.364471 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.364416    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:22.399684 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.399648    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea93397-c1e0-4687-8b0d-fa29856952bb-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76\" (UID: \"eea93397-c1e0-4687-8b0d-fa29856952bb\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"
Sep 05 03:14:22.399834 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.399743    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5485\" (UniqueName: \"kubernetes.io/projected/eea93397-c1e0-4687-8b0d-fa29856952bb-kube-api-access-q5485\") pod \"e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76\" (UID: \"eea93397-c1e0-4687-8b0d-fa29856952bb\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"
Sep 05 03:14:22.500341 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.500296    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea93397-c1e0-4687-8b0d-fa29856952bb-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76\" (UID: \"eea93397-c1e0-4687-8b0d-fa29856952bb\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"
Sep 05 03:14:22.500534 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.500370    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5485\" (UniqueName: \"kubernetes.io/projected/eea93397-c1e0-4687-8b0d-fa29856952bb-kube-api-access-q5485\") pod \"e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76\" (UID: \"eea93397-c1e0-4687-8b0d-fa29856952bb\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"
Sep 05 03:14:22.500761 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.500736    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea93397-c1e0-4687-8b0d-fa29856952bb-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76\" (UID: \"eea93397-c1e0-4687-8b0d-fa29856952bb\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"
Sep 05 03:14:22.508949 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.508923    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5485\" (UniqueName: \"kubernetes.io/projected/eea93397-c1e0-4687-8b0d-fa29856952bb-kube-api-access-q5485\") pod \"e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76\" (UID: \"eea93397-c1e0-4687-8b0d-fa29856952bb\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"
Sep 05 03:14:22.593927 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.593886    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:22.633133 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.633104    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"
Sep 05 03:14:22.911570 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:22.911473    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"]
Sep 05 03:14:22.914501 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:14:22.914471    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea93397_c1e0_4687_8b0d_fa29856952bb.slice/crio-9233fdecee42588ed278f2a3fd76c08b448b308b2f4ef3eaf16c4b6986299a1d WatchSource:0}: Error finding container 9233fdecee42588ed278f2a3fd76c08b448b308b2f4ef3eaf16c4b6986299a1d: Status 404 returned error can't find the container with id 9233fdecee42588ed278f2a3fd76c08b448b308b2f4ef3eaf16c4b6986299a1d
Sep 05 03:14:23.225862 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:23.225745    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 05 03:14:23.567659 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:23.567627    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76" event={"ID":"eea93397-c1e0-4687-8b0d-fa29856952bb","Type":"ContainerStarted","Data":"7cfec888d7ddcdd920f1c68dab96e10e73da16a38319508e81dcb75ea48e0e57"}
Sep 05 03:14:23.568118 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:23.567689    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76" event={"ID":"eea93397-c1e0-4687-8b0d-fa29856952bb","Type":"ContainerStarted","Data":"9233fdecee42588ed278f2a3fd76c08b448b308b2f4ef3eaf16c4b6986299a1d"}
Sep 05 03:14:25.253036 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:25.252999    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"]
Sep 05 03:14:25.253869 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:25.253651    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:25.348479 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:25.348371    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:25.582486 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:25.582424    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76" podUID="eea93397-c1e0-4687-8b0d-fa29856952bb" containerName="storage-initializer" containerID="cri-o://7cfec888d7ddcdd920f1c68dab96e10e73da16a38319508e81dcb75ea48e0e57" gracePeriod=30
Sep 05 03:14:26.307290 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:26.307248    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:26.407179 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:26.407132    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:26.507987 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:26.507949    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:27.459808 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.459780    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"
Sep 05 03:14:27.547995 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.547910    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5485\" (UniqueName: \"kubernetes.io/projected/eea93397-c1e0-4687-8b0d-fa29856952bb-kube-api-access-q5485\") pod \"eea93397-c1e0-4687-8b0d-fa29856952bb\" (UID: \"eea93397-c1e0-4687-8b0d-fa29856952bb\") "
Sep 05 03:14:27.547995 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.547975    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea93397-c1e0-4687-8b0d-fa29856952bb-kserve-provision-location\") pod \"eea93397-c1e0-4687-8b0d-fa29856952bb\" (UID: \"eea93397-c1e0-4687-8b0d-fa29856952bb\") "
Sep 05 03:14:27.550263 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.550236    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea93397-c1e0-4687-8b0d-fa29856952bb-kube-api-access-q5485" (OuterVolumeSpecName: "kube-api-access-q5485") pod "eea93397-c1e0-4687-8b0d-fa29856952bb" (UID: "eea93397-c1e0-4687-8b0d-fa29856952bb"). InnerVolumeSpecName "kube-api-access-q5485". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:14:27.596575 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.596536    2557 generic.go:356] "Generic (PLEG): container finished" podID="eea93397-c1e0-4687-8b0d-fa29856952bb" containerID="7cfec888d7ddcdd920f1c68dab96e10e73da16a38319508e81dcb75ea48e0e57" exitCode=0
Sep 05 03:14:27.596759 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.596604    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"
Sep 05 03:14:27.596759 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.596629    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76" event={"ID":"eea93397-c1e0-4687-8b0d-fa29856952bb","Type":"ContainerDied","Data":"7cfec888d7ddcdd920f1c68dab96e10e73da16a38319508e81dcb75ea48e0e57"}
Sep 05 03:14:27.596759 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.596694    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76" event={"ID":"eea93397-c1e0-4687-8b0d-fa29856952bb","Type":"ContainerDied","Data":"9233fdecee42588ed278f2a3fd76c08b448b308b2f4ef3eaf16c4b6986299a1d"}
Sep 05 03:14:27.596759 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.596719    2557 scope.go:117] "RemoveContainer" containerID="7cfec888d7ddcdd920f1c68dab96e10e73da16a38319508e81dcb75ea48e0e57"
Sep 05 03:14:27.608080 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.608057    2557 scope.go:117] "RemoveContainer" containerID="7cfec888d7ddcdd920f1c68dab96e10e73da16a38319508e81dcb75ea48e0e57"
Sep 05 03:14:27.608385 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:14:27.608360    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cfec888d7ddcdd920f1c68dab96e10e73da16a38319508e81dcb75ea48e0e57\": container with ID starting with 7cfec888d7ddcdd920f1c68dab96e10e73da16a38319508e81dcb75ea48e0e57 not found: ID does not exist" containerID="7cfec888d7ddcdd920f1c68dab96e10e73da16a38319508e81dcb75ea48e0e57"
Sep 05 03:14:27.608497 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.608397    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cfec888d7ddcdd920f1c68dab96e10e73da16a38319508e81dcb75ea48e0e57"} err="failed to get container status \"7cfec888d7ddcdd920f1c68dab96e10e73da16a38319508e81dcb75ea48e0e57\": rpc error: code = NotFound desc = could not find container \"7cfec888d7ddcdd920f1c68dab96e10e73da16a38319508e81dcb75ea48e0e57\": container with ID starting with 7cfec888d7ddcdd920f1c68dab96e10e73da16a38319508e81dcb75ea48e0e57 not found: ID does not exist"
Sep 05 03:14:27.622379 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.622345    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea93397-c1e0-4687-8b0d-fa29856952bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eea93397-c1e0-4687-8b0d-fa29856952bb" (UID: "eea93397-c1e0-4687-8b0d-fa29856952bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:14:27.649214 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.649181    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5485\" (UniqueName: \"kubernetes.io/projected/eea93397-c1e0-4687-8b0d-fa29856952bb-kube-api-access-q5485\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:14:27.649214 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.649214    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea93397-c1e0-4687-8b0d-fa29856952bb-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:14:27.937831 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.937794    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"]
Sep 05 03:14:27.942343 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:27.942299    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6cdc44c76c-p7b76"]
Sep 05 03:14:28.993170 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:28.993131    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eea93397-c1e0-4687-8b0d-fa29856952bb" path="/var/lib/kubelet/pods/eea93397-c1e0-4687-8b0d-fa29856952bb/volumes"
Sep 05 03:14:31.389508 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:31.389456    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:33.225634 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:33.225576    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 05 03:14:37.025360 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.025327    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"]
Sep 05 03:14:37.027610 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.025712    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea93397-c1e0-4687-8b0d-fa29856952bb" containerName="storage-initializer"
Sep 05 03:14:37.027610 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.025727    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea93397-c1e0-4687-8b0d-fa29856952bb" containerName="storage-initializer"
Sep 05 03:14:37.027610 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.025843    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="eea93397-c1e0-4687-8b0d-fa29856952bb" containerName="storage-initializer"
Sep 05 03:14:37.029210 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.029192    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.031874 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.031846    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 05 03:14:37.032035 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.031913    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 05 03:14:37.032035 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.031970    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-rpdns\""
Sep 05 03:14:37.032172 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.032031    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 05 03:14:37.032172 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.032092    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:14:37.032313 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.032294    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 05 03:14:37.040712 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.040689    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"]
Sep 05 03:14:37.100890 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.100850    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:37.137030 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.136993    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-home\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.137189 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.137042    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-kserve-provision-location\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.137189 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.137092    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-tmp-dir\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.137189 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.137158    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-dshm\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.137301 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.137215    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d17008-b031-4753-8f92-05205a11fc61-tls-certs\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.137301 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.137275    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d17008-b031-4753-8f92-05205a11fc61-cabundle-cert\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.137368 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.137313    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-model-cache\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.137368 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.137353    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz5vb\" (UniqueName: \"kubernetes.io/projected/c1d17008-b031-4753-8f92-05205a11fc61-kube-api-access-bz5vb\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.238727 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.238691    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-tmp-dir\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.238918 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.238737    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-dshm\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.238918 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.238775    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d17008-b031-4753-8f92-05205a11fc61-tls-certs\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.238918 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.238813    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d17008-b031-4753-8f92-05205a11fc61-cabundle-cert\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.238918 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.238842    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-model-cache\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.238918 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.238879    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz5vb\" (UniqueName: \"kubernetes.io/projected/c1d17008-b031-4753-8f92-05205a11fc61-kube-api-access-bz5vb\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.239210 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.238939    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-home\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.239210 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.238974    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-kserve-provision-location\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.239384 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.239356    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-kserve-provision-location\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.239733 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.239708    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d17008-b031-4753-8f92-05205a11fc61-cabundle-cert\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.239846 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.239735    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-home\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.239846 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.239788    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-model-cache\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.239975 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.239946    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-tmp-dir\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.242355 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.242304    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-dshm\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.242955 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.242931    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d17008-b031-4753-8f92-05205a11fc61-tls-certs\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.250628 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.250531    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz5vb\" (UniqueName: \"kubernetes.io/projected/c1d17008-b031-4753-8f92-05205a11fc61-kube-api-access-bz5vb\") pod \"stop-feature-test-kserve-6cc696dc5d-8xcct\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.284150 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.284025    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:37.340892 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.340849    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:37.630364 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.630338    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"]
Sep 05 03:14:37.633135 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:14:37.633105    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d17008_b031_4753_8f92_05205a11fc61.slice/crio-96704a01d504dfaba9bff936ff71dba96320b826629f837969229923549afb1d WatchSource:0}: Error finding container 96704a01d504dfaba9bff936ff71dba96320b826629f837969229923549afb1d: Status 404 returned error can't find the container with id 96704a01d504dfaba9bff936ff71dba96320b826629f837969229923549afb1d
Sep 05 03:14:37.658331 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:37.658291    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" event={"ID":"c1d17008-b031-4753-8f92-05205a11fc61","Type":"ContainerStarted","Data":"96704a01d504dfaba9bff936ff71dba96320b826629f837969229923549afb1d"}
Sep 05 03:14:38.669475 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:38.669418    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" event={"ID":"c1d17008-b031-4753-8f92-05205a11fc61","Type":"ContainerStarted","Data":"068602bf3c4c781cda871806b7eef3a5cf56db6bfa3da947c682d4d9270ec9e5"}
Sep 05 03:14:40.684742 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:40.684697    2557 generic.go:356] "Generic (PLEG): container finished" podID="c1d17008-b031-4753-8f92-05205a11fc61" containerID="068602bf3c4c781cda871806b7eef3a5cf56db6bfa3da947c682d4d9270ec9e5" exitCode=0
Sep 05 03:14:40.685250 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:40.684767    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" event={"ID":"c1d17008-b031-4753-8f92-05205a11fc61","Type":"ContainerDied","Data":"068602bf3c4c781cda871806b7eef3a5cf56db6bfa3da947c682d4d9270ec9e5"}
Sep 05 03:14:41.694384 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:41.694347    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" event={"ID":"c1d17008-b031-4753-8f92-05205a11fc61","Type":"ContainerStarted","Data":"6126f1bd4d85e6685e334e5ba7802f6fced38474c384e286909a58b9aa6a17a5"}
Sep 05 03:14:41.716495 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:41.716401    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" podStartSLOduration=2.271313187 podStartE2EDuration="4.716379775s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.445066588s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:14:41.714263346 +0000 UTC m=+3385.269419671" watchObservedRunningTime="2026-09-05 03:14:41.716379775 +0000 UTC m=+3385.271536088"
Sep 05 03:14:42.341783 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:42.341740    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:43.225731 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:43.225684    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 05 03:14:47.341334 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:47.341277    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:14:47.342981 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:47.342947    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 05 03:14:51.939694 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:51.939648    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:14:53.226309 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:53.226250    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 05 03:14:57.342061 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:14:57.342014    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 05 03:15:03.226188 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:03.226125    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 05 03:15:07.341899 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:07.341844    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 05 03:15:13.226603 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:13.226488    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 05 03:15:17.341867 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:17.341817    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 05 03:15:19.363799 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:19.363755    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:15:23.225805 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:23.225756    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 05 03:15:27.342264 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:27.342209    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 05 03:15:30.428708 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:30.428664    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:15:33.236309 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:33.236272    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:15:33.244460 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:33.244416    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:15:37.342125 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:37.342070    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 05 03:15:38.336290 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:38.336249    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"]
Sep 05 03:15:38.343578 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:38.343546    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:15:38.405818 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:38.405780    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:15:47.341864 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:47.341822    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 05 03:15:53.365332 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:53.365285    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main" containerID="cri-o://7ca00187e6628afeb267a123ca728fc2331828a4b46baf408cb4594512ec984d" gracePeriod=45
Sep 05 03:15:54.237096 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.237039    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 05 03:15:54.425369 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.425340    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:15:54.520807 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.520715    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-kserve-provision-location\") pod \"7946ec2b-ac41-4017-a743-e19ada9d7709\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") "
Sep 05 03:15:54.520807 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.520762    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-tmp-dir\") pod \"7946ec2b-ac41-4017-a743-e19ada9d7709\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") "
Sep 05 03:15:54.521039 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.520830    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-dshm\") pod \"7946ec2b-ac41-4017-a743-e19ada9d7709\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") "
Sep 05 03:15:54.521039 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.520854    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-home\") pod \"7946ec2b-ac41-4017-a743-e19ada9d7709\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") "
Sep 05 03:15:54.521039 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.520883    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-model-cache\") pod \"7946ec2b-ac41-4017-a743-e19ada9d7709\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") "
Sep 05 03:15:54.521039 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.520914    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxzpt\" (UniqueName: \"kubernetes.io/projected/7946ec2b-ac41-4017-a743-e19ada9d7709-kube-api-access-wxzpt\") pod \"7946ec2b-ac41-4017-a743-e19ada9d7709\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") "
Sep 05 03:15:54.521039 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.520945    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7946ec2b-ac41-4017-a743-e19ada9d7709-tls-certs\") pod \"7946ec2b-ac41-4017-a743-e19ada9d7709\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") "
Sep 05 03:15:54.521039 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.520970    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7946ec2b-ac41-4017-a743-e19ada9d7709-cabundle-cert\") pod \"7946ec2b-ac41-4017-a743-e19ada9d7709\" (UID: \"7946ec2b-ac41-4017-a743-e19ada9d7709\") "
Sep 05 03:15:54.521715 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.521687    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7946ec2b-ac41-4017-a743-e19ada9d7709-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7946ec2b-ac41-4017-a743-e19ada9d7709" (UID: "7946ec2b-ac41-4017-a743-e19ada9d7709"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:15:54.522052 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.522026    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-model-cache" (OuterVolumeSpecName: "model-cache") pod "7946ec2b-ac41-4017-a743-e19ada9d7709" (UID: "7946ec2b-ac41-4017-a743-e19ada9d7709"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:15:54.522827 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.522781    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-home" (OuterVolumeSpecName: "home") pod "7946ec2b-ac41-4017-a743-e19ada9d7709" (UID: "7946ec2b-ac41-4017-a743-e19ada9d7709"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:15:54.523707 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.523672    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7946ec2b-ac41-4017-a743-e19ada9d7709-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7946ec2b-ac41-4017-a743-e19ada9d7709" (UID: "7946ec2b-ac41-4017-a743-e19ada9d7709"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:15:54.523707 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.523682    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7946ec2b-ac41-4017-a743-e19ada9d7709-kube-api-access-wxzpt" (OuterVolumeSpecName: "kube-api-access-wxzpt") pod "7946ec2b-ac41-4017-a743-e19ada9d7709" (UID: "7946ec2b-ac41-4017-a743-e19ada9d7709"). InnerVolumeSpecName "kube-api-access-wxzpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:15:54.523972 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.523954    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-dshm" (OuterVolumeSpecName: "dshm") pod "7946ec2b-ac41-4017-a743-e19ada9d7709" (UID: "7946ec2b-ac41-4017-a743-e19ada9d7709"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:15:54.533921 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.533888    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7946ec2b-ac41-4017-a743-e19ada9d7709" (UID: "7946ec2b-ac41-4017-a743-e19ada9d7709"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:15:54.547553 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.547516    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7946ec2b-ac41-4017-a743-e19ada9d7709" (UID: "7946ec2b-ac41-4017-a743-e19ada9d7709"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:15:54.622133 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.622089    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:15:54.622133 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.622126    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:15:54.622133 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.622137    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:15:54.622133 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.622145    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:15:54.622555 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.622154    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7946ec2b-ac41-4017-a743-e19ada9d7709-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:15:54.622555 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.622162    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxzpt\" (UniqueName: \"kubernetes.io/projected/7946ec2b-ac41-4017-a743-e19ada9d7709-kube-api-access-wxzpt\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:15:54.622555 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.622183    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7946ec2b-ac41-4017-a743-e19ada9d7709-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:15:54.622555 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:54.622193    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7946ec2b-ac41-4017-a743-e19ada9d7709-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:15:55.157023 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:55.156989    2557 generic.go:356] "Generic (PLEG): container finished" podID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerID="7ca00187e6628afeb267a123ca728fc2331828a4b46baf408cb4594512ec984d" exitCode=0
Sep 05 03:15:55.157194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:55.157039    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" event={"ID":"7946ec2b-ac41-4017-a743-e19ada9d7709","Type":"ContainerDied","Data":"7ca00187e6628afeb267a123ca728fc2331828a4b46baf408cb4594512ec984d"}
Sep 05 03:15:55.157194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:55.157120    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt" event={"ID":"7946ec2b-ac41-4017-a743-e19ada9d7709","Type":"ContainerDied","Data":"07411336a8d99e2228ed1cf80a9fe1d3aebe01b9fef3131e920807ca1f16c64c"}
Sep 05 03:15:55.157194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:55.157058    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"
Sep 05 03:15:55.157194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:55.157155    2557 scope.go:117] "RemoveContainer" containerID="7ca00187e6628afeb267a123ca728fc2331828a4b46baf408cb4594512ec984d"
Sep 05 03:15:55.166292 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:55.166267    2557 scope.go:117] "RemoveContainer" containerID="65ba8e6d335e3e0bf49bc4ed30d2ead62d3029bb560ee5deff9ff47d45ff771b"
Sep 05 03:15:55.174123 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:55.174106    2557 scope.go:117] "RemoveContainer" containerID="7ca00187e6628afeb267a123ca728fc2331828a4b46baf408cb4594512ec984d"
Sep 05 03:15:55.174391 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:15:55.174360    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca00187e6628afeb267a123ca728fc2331828a4b46baf408cb4594512ec984d\": container with ID starting with 7ca00187e6628afeb267a123ca728fc2331828a4b46baf408cb4594512ec984d not found: ID does not exist" containerID="7ca00187e6628afeb267a123ca728fc2331828a4b46baf408cb4594512ec984d"
Sep 05 03:15:55.174528 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:55.174392    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca00187e6628afeb267a123ca728fc2331828a4b46baf408cb4594512ec984d"} err="failed to get container status \"7ca00187e6628afeb267a123ca728fc2331828a4b46baf408cb4594512ec984d\": rpc error: code = NotFound desc = could not find container \"7ca00187e6628afeb267a123ca728fc2331828a4b46baf408cb4594512ec984d\": container with ID starting with 7ca00187e6628afeb267a123ca728fc2331828a4b46baf408cb4594512ec984d not found: ID does not exist"
Sep 05 03:15:55.174528 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:55.174410    2557 scope.go:117] "RemoveContainer" containerID="65ba8e6d335e3e0bf49bc4ed30d2ead62d3029bb560ee5deff9ff47d45ff771b"
Sep 05 03:15:55.174689 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:15:55.174670    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ba8e6d335e3e0bf49bc4ed30d2ead62d3029bb560ee5deff9ff47d45ff771b\": container with ID starting with 65ba8e6d335e3e0bf49bc4ed30d2ead62d3029bb560ee5deff9ff47d45ff771b not found: ID does not exist" containerID="65ba8e6d335e3e0bf49bc4ed30d2ead62d3029bb560ee5deff9ff47d45ff771b"
Sep 05 03:15:55.174758 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:55.174692    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ba8e6d335e3e0bf49bc4ed30d2ead62d3029bb560ee5deff9ff47d45ff771b"} err="failed to get container status \"65ba8e6d335e3e0bf49bc4ed30d2ead62d3029bb560ee5deff9ff47d45ff771b\": rpc error: code = NotFound desc = could not find container \"65ba8e6d335e3e0bf49bc4ed30d2ead62d3029bb560ee5deff9ff47d45ff771b\": container with ID starting with 65ba8e6d335e3e0bf49bc4ed30d2ead62d3029bb560ee5deff9ff47d45ff771b not found: ID does not exist"
Sep 05 03:15:55.182763 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:55.181869    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"]
Sep 05 03:15:55.187818 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:55.187782    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-597d4f555f-nbcqt"]
Sep 05 03:15:56.992360 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:56.992313    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" path="/var/lib/kubelet/pods/7946ec2b-ac41-4017-a743-e19ada9d7709/volumes"
Sep 05 03:15:57.342346 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:15:57.342290    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 05 03:16:07.341353 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:07.341301    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 05 03:16:17.341733 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:17.341690    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 05 03:16:25.808112 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:25.808073    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:16:27.186484 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:27.186427    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:16:27.341640 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:27.341595    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 05 03:16:28.466950 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.466909    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:16:28.568321 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.568273    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:16:28.702080 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.702047    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"]
Sep 05 03:16:28.702526 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.702503    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="storage-initializer"
Sep 05 03:16:28.702526 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.702526    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="storage-initializer"
Sep 05 03:16:28.702711 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.702554    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main"
Sep 05 03:16:28.702711 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.702561    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main"
Sep 05 03:16:28.702711 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.702631    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7946ec2b-ac41-4017-a743-e19ada9d7709" containerName="main"
Sep 05 03:16:28.708844 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.708819    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.711370 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.711343    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-hbz8s\""
Sep 05 03:16:28.711615 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.711375    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 05 03:16:28.711716 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.711381    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 05 03:16:28.712051 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.712030    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 05 03:16:28.720566 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.720509    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"]
Sep 05 03:16:28.737629 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.737575    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:16:28.830492 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.830432    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:16:28.838635 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.838599    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.838805 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.838694    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.838805 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.838749    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.838934 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.838849    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.838934 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.838889    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.839047 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.838963    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.839047 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.839017    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.839142 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.839114    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b9d643-fbbf-43c6-940d-621a39c648b6-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.839195 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.839155    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsjpj\" (UniqueName: \"kubernetes.io/projected/61b9d643-fbbf-43c6-940d-621a39c648b6-kube-api-access-fsjpj\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.940648 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.940605    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.940648 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.940653    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.940906 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.940696    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.940906 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.940726    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.940906 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.940747    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.940906 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.940783    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.940906 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.940812    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.940906 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.940851    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b9d643-fbbf-43c6-940d-621a39c648b6-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.940906 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.940886    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsjpj\" (UniqueName: \"kubernetes.io/projected/61b9d643-fbbf-43c6-940d-621a39c648b6-kube-api-access-fsjpj\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.941261 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.941188    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.941261 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.941206    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.941261 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.941220    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.941550 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.941528    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.941930 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.941899    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b9d643-fbbf-43c6-940d-621a39c648b6-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.943376 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.943350    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.943586 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.943564    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.948110 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.948082    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:28.948285 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:28.948264    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsjpj\" (UniqueName: \"kubernetes.io/projected/61b9d643-fbbf-43c6-940d-621a39c648b6-kube-api-access-fsjpj\") pod \"router-gateway-1-openshift-default-56bd58cf4d-8r6r2\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:29.021204 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:29.021129    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:29.311288 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:29.311249    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"]
Sep 05 03:16:29.313219 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:16:29.313186    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b9d643_fbbf_43c6_940d_621a39c648b6.slice/crio-97b1f1a68a532a774fac0b9397a527f81284cf27c1f5d25d6dad00925679ec52 WatchSource:0}: Error finding container 97b1f1a68a532a774fac0b9397a527f81284cf27c1f5d25d6dad00925679ec52: Status 404 returned error can't find the container with id 97b1f1a68a532a774fac0b9397a527f81284cf27c1f5d25d6dad00925679ec52
Sep 05 03:16:29.315973 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:29.315931    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:16:29.316087 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:29.316021    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:16:29.316087 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:29.316067    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:16:29.349497 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:29.349467    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2" event={"ID":"61b9d643-fbbf-43c6-940d-621a39c648b6","Type":"ContainerStarted","Data":"97b1f1a68a532a774fac0b9397a527f81284cf27c1f5d25d6dad00925679ec52"}
Sep 05 03:16:30.356146 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:30.356101    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2" event={"ID":"61b9d643-fbbf-43c6-940d-621a39c648b6","Type":"ContainerStarted","Data":"a59a4a621df7085087f4bb2f10419755edac7c675a6a3286425b3b4ef723a7d3"}
Sep 05 03:16:30.377818 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:30.377758    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2" podStartSLOduration=2.377741402 podStartE2EDuration="2.377741402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:16:30.376999572 +0000 UTC m=+3493.932155882" watchObservedRunningTime="2026-09-05 03:16:30.377741402 +0000 UTC m=+3493.932897710"
Sep 05 03:16:31.022273 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:31.022235    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:31.027510 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:31.027475    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:31.365862 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:31.365821    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:31.367452 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:31.367415    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:31.662307 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:31.662269    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:16:35.454163 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:35.454119    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:16:37.362304 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:37.362262    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:16:37.372849 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:37.372811    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:16:41.232233 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:41.232150    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:16:41.302143 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:41.302031    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:16:41.427498 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:41.427463    2557 generic.go:356] "Generic (PLEG): container finished" podID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerID="a59a4a621df7085087f4bb2f10419755edac7c675a6a3286425b3b4ef723a7d3" exitCode=0
Sep 05 03:16:41.427669 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:41.427516    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2" event={"ID":"61b9d643-fbbf-43c6-940d-621a39c648b6","Type":"ContainerDied","Data":"a59a4a621df7085087f4bb2f10419755edac7c675a6a3286425b3b4ef723a7d3"}
Sep 05 03:16:41.427926 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:41.427910    2557 scope.go:117] "RemoveContainer" containerID="a59a4a621df7085087f4bb2f10419755edac7c675a6a3286425b3b4ef723a7d3"
Sep 05 03:16:41.428874 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:41.428819    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:16:41.428996 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:41.428910    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:16:41.428996 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:41.428957    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:16:42.434974 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:42.434928    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2" event={"ID":"61b9d643-fbbf-43c6-940d-621a39c648b6","Type":"ContainerStarted","Data":"81a3411fb5cdae972e1ad8069371b4b1033380c1af49e70c9cb4e84407fa7ed7"}
Sep 05 03:16:43.021849 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:43.021807    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:43.023729 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:43.023690    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2" podUID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Sep 05 03:16:44.021770 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:44.021723    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2" podUID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Sep 05 03:16:45.021533 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:45.021491    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2" podUID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Sep 05 03:16:46.022036 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:46.021992    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2" podUID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Sep 05 03:16:46.366657 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:46.366602    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:47.022145 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:47.022104    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2" podUID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Sep 05 03:16:48.022476 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:48.022407    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2" podUID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Sep 05 03:16:49.026492 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:49.026457    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:16:49.027314 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:16:49.027295    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:17:01.018468 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.018414    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"]
Sep 05 03:17:01.021921 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.021898    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.023985 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.023963    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 05 03:17:01.024352 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.024332    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-lmnqm\""
Sep 05 03:17:01.024460 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.024394    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:17:01.024460 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.024400    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 05 03:17:01.034835 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.034810    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"]
Sep 05 03:17:01.045277 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.045199    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:17:01.132214 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.132176    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-kserve-provision-location\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.132214 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.132214    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce67741-fb12-4f40-b8e0-0ccb8931141a-tls-certs\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.132415 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.132233    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-tmp-dir\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.132415 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.132336    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-home\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.132415 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.132369    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gf77\" (UniqueName: \"kubernetes.io/projected/fce67741-fb12-4f40-b8e0-0ccb8931141a-kube-api-access-2gf77\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.132415 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.132391    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-dshm\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.132578 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.132420    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-model-cache\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.132578 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.132498    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce67741-fb12-4f40-b8e0-0ccb8931141a-cabundle-cert\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.233597 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.233558    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-home\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.233597 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.233599    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gf77\" (UniqueName: \"kubernetes.io/projected/fce67741-fb12-4f40-b8e0-0ccb8931141a-kube-api-access-2gf77\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.233833 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.233624    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-dshm\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.233833 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.233648    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-model-cache\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.233833 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.233728    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce67741-fb12-4f40-b8e0-0ccb8931141a-cabundle-cert\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.233833 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.233763    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-kserve-provision-location\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.233833 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.233789    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce67741-fb12-4f40-b8e0-0ccb8931141a-tls-certs\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.233833 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.233815    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-tmp-dir\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.234103 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.234086    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-home\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.234163 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.234119    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-kserve-provision-location\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.234227 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.234208    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-tmp-dir\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.234290 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.234235    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-model-cache\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.234674 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.234651    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce67741-fb12-4f40-b8e0-0ccb8931141a-cabundle-cert\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.235993 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.235974    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-dshm\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.236264 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.236244    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce67741-fb12-4f40-b8e0-0ccb8931141a-tls-certs\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.241453 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.241418    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gf77\" (UniqueName: \"kubernetes.io/projected/fce67741-fb12-4f40-b8e0-0ccb8931141a-kube-api-access-2gf77\") pod \"router-with-refs-test-kserve-79c695dc8c-rf8m8\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.332769 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.332729    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:01.513750 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.513710    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:17:01.593913 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.593821    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:17:01.616684 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:01.616655    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"]
Sep 05 03:17:01.619049 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:17:01.619019    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce67741_fb12_4f40_b8e0_0ccb8931141a.slice/crio-da6ffe51c6eee3ccf2db2b2592e2eec4164b64a7666f6840dec9d7bea0f59ebd WatchSource:0}: Error finding container da6ffe51c6eee3ccf2db2b2592e2eec4164b64a7666f6840dec9d7bea0f59ebd: Status 404 returned error can't find the container with id da6ffe51c6eee3ccf2db2b2592e2eec4164b64a7666f6840dec9d7bea0f59ebd
Sep 05 03:17:02.153187 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:02.153146    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:17:02.558290 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:02.558182    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" event={"ID":"fce67741-fb12-4f40-b8e0-0ccb8931141a","Type":"ContainerStarted","Data":"40f35e6ef11f15424733f9dba4759b31ada9596e689d2a6984a4cd2dad1c89cb"}
Sep 05 03:17:02.558486 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:02.558289    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" event={"ID":"fce67741-fb12-4f40-b8e0-0ccb8931141a","Type":"ContainerStarted","Data":"da6ffe51c6eee3ccf2db2b2592e2eec4164b64a7666f6840dec9d7bea0f59ebd"}
Sep 05 03:17:04.573483 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:04.573415    2557 generic.go:356] "Generic (PLEG): container finished" podID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerID="40f35e6ef11f15424733f9dba4759b31ada9596e689d2a6984a4cd2dad1c89cb" exitCode=0
Sep 05 03:17:04.573950 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:04.573481    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" event={"ID":"fce67741-fb12-4f40-b8e0-0ccb8931141a","Type":"ContainerDied","Data":"40f35e6ef11f15424733f9dba4759b31ada9596e689d2a6984a4cd2dad1c89cb"}
Sep 05 03:17:05.581876 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:05.581836    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" event={"ID":"fce67741-fb12-4f40-b8e0-0ccb8931141a","Type":"ContainerStarted","Data":"0f2c99fe39a7d752195b93c2949dfb101084f19f2b07ebe858436fdd424a63d7"}
Sep 05 03:17:05.606151 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:05.606093    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" podStartSLOduration=3.204460437 podStartE2EDuration="5.606076856s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.401616419s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:17:05.603559971 +0000 UTC m=+3529.158716306" watchObservedRunningTime="2026-09-05 03:17:05.606076856 +0000 UTC m=+3529.161233165"
Sep 05 03:17:06.332829 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:06.332790    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:10.389091 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:10.389028    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:17:11.333205 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:11.333162    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:17:11.334877 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:11.334845    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 05 03:17:12.205580 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:12.205533    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"]
Sep 05 03:17:12.231429 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:12.231390    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:17:12.292996 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:12.292957    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:17:12.387310 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:12.387272    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:17:21.333829 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:21.333778    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 05 03:17:27.232660 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:27.232584    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main" containerID="cri-o://6126f1bd4d85e6685e334e5ba7802f6fced38474c384e286909a58b9aa6a17a5" gracePeriod=45
Sep 05 03:17:27.364028 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:27.363978    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 05 03:17:27.988342 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:27.988316    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:17:28.085151 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.085111    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-tmp-dir\") pod \"c1d17008-b031-4753-8f92-05205a11fc61\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") "
Sep 05 03:17:28.085322 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.085202    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d17008-b031-4753-8f92-05205a11fc61-tls-certs\") pod \"c1d17008-b031-4753-8f92-05205a11fc61\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") "
Sep 05 03:17:28.085322 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.085235    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz5vb\" (UniqueName: \"kubernetes.io/projected/c1d17008-b031-4753-8f92-05205a11fc61-kube-api-access-bz5vb\") pod \"c1d17008-b031-4753-8f92-05205a11fc61\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") "
Sep 05 03:17:28.085322 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.085277    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-dshm\") pod \"c1d17008-b031-4753-8f92-05205a11fc61\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") "
Sep 05 03:17:28.085322 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.085320    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-kserve-provision-location\") pod \"c1d17008-b031-4753-8f92-05205a11fc61\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") "
Sep 05 03:17:28.085571 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.085356    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-home\") pod \"c1d17008-b031-4753-8f92-05205a11fc61\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") "
Sep 05 03:17:28.085571 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.085380    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d17008-b031-4753-8f92-05205a11fc61-cabundle-cert\") pod \"c1d17008-b031-4753-8f92-05205a11fc61\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") "
Sep 05 03:17:28.085571 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.085480    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-model-cache\") pod \"c1d17008-b031-4753-8f92-05205a11fc61\" (UID: \"c1d17008-b031-4753-8f92-05205a11fc61\") "
Sep 05 03:17:28.086550 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.086519    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1d17008-b031-4753-8f92-05205a11fc61" (UID: "c1d17008-b031-4753-8f92-05205a11fc61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:17:28.086744 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.086706    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d17008-b031-4753-8f92-05205a11fc61-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c1d17008-b031-4753-8f92-05205a11fc61" (UID: "c1d17008-b031-4753-8f92-05205a11fc61"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:17:28.088129 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.087764    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-home" (OuterVolumeSpecName: "home") pod "c1d17008-b031-4753-8f92-05205a11fc61" (UID: "c1d17008-b031-4753-8f92-05205a11fc61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:17:28.088247 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.088117    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-dshm" (OuterVolumeSpecName: "dshm") pod "c1d17008-b031-4753-8f92-05205a11fc61" (UID: "c1d17008-b031-4753-8f92-05205a11fc61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:17:28.088247 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.088180    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d17008-b031-4753-8f92-05205a11fc61-kube-api-access-bz5vb" (OuterVolumeSpecName: "kube-api-access-bz5vb") pod "c1d17008-b031-4753-8f92-05205a11fc61" (UID: "c1d17008-b031-4753-8f92-05205a11fc61"). InnerVolumeSpecName "kube-api-access-bz5vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:17:28.088247 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.088192    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d17008-b031-4753-8f92-05205a11fc61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1d17008-b031-4753-8f92-05205a11fc61" (UID: "c1d17008-b031-4753-8f92-05205a11fc61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:17:28.098927 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.098894    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1d17008-b031-4753-8f92-05205a11fc61" (UID: "c1d17008-b031-4753-8f92-05205a11fc61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:17:28.100631 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.100604    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1d17008-b031-4753-8f92-05205a11fc61" (UID: "c1d17008-b031-4753-8f92-05205a11fc61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:17:28.187456 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.187411    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:17:28.187604 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.187475    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:17:28.187604 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.187492    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d17008-b031-4753-8f92-05205a11fc61-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:17:28.187604 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.187505    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz5vb\" (UniqueName: \"kubernetes.io/projected/c1d17008-b031-4753-8f92-05205a11fc61-kube-api-access-bz5vb\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:17:28.187604 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.187521    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:17:28.187604 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.187536    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:17:28.187604 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.187550    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d17008-b031-4753-8f92-05205a11fc61-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:17:28.187604 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.187563    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1d17008-b031-4753-8f92-05205a11fc61-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:17:28.728502 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.728465    2557 generic.go:356] "Generic (PLEG): container finished" podID="c1d17008-b031-4753-8f92-05205a11fc61" containerID="6126f1bd4d85e6685e334e5ba7802f6fced38474c384e286909a58b9aa6a17a5" exitCode=0
Sep 05 03:17:28.729018 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.728525    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"
Sep 05 03:17:28.729018 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.728560    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" event={"ID":"c1d17008-b031-4753-8f92-05205a11fc61","Type":"ContainerDied","Data":"6126f1bd4d85e6685e334e5ba7802f6fced38474c384e286909a58b9aa6a17a5"}
Sep 05 03:17:28.729018 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.728631    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct" event={"ID":"c1d17008-b031-4753-8f92-05205a11fc61","Type":"ContainerDied","Data":"96704a01d504dfaba9bff936ff71dba96320b826629f837969229923549afb1d"}
Sep 05 03:17:28.729018 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.728665    2557 scope.go:117] "RemoveContainer" containerID="6126f1bd4d85e6685e334e5ba7802f6fced38474c384e286909a58b9aa6a17a5"
Sep 05 03:17:28.739684 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.739664    2557 scope.go:117] "RemoveContainer" containerID="068602bf3c4c781cda871806b7eef3a5cf56db6bfa3da947c682d4d9270ec9e5"
Sep 05 03:17:28.747777 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.747760    2557 scope.go:117] "RemoveContainer" containerID="6126f1bd4d85e6685e334e5ba7802f6fced38474c384e286909a58b9aa6a17a5"
Sep 05 03:17:28.748088 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:17:28.748063    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6126f1bd4d85e6685e334e5ba7802f6fced38474c384e286909a58b9aa6a17a5\": container with ID starting with 6126f1bd4d85e6685e334e5ba7802f6fced38474c384e286909a58b9aa6a17a5 not found: ID does not exist" containerID="6126f1bd4d85e6685e334e5ba7802f6fced38474c384e286909a58b9aa6a17a5"
Sep 05 03:17:28.748166 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.748101    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6126f1bd4d85e6685e334e5ba7802f6fced38474c384e286909a58b9aa6a17a5"} err="failed to get container status \"6126f1bd4d85e6685e334e5ba7802f6fced38474c384e286909a58b9aa6a17a5\": rpc error: code = NotFound desc = could not find container \"6126f1bd4d85e6685e334e5ba7802f6fced38474c384e286909a58b9aa6a17a5\": container with ID starting with 6126f1bd4d85e6685e334e5ba7802f6fced38474c384e286909a58b9aa6a17a5 not found: ID does not exist"
Sep 05 03:17:28.748166 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.748125    2557 scope.go:117] "RemoveContainer" containerID="068602bf3c4c781cda871806b7eef3a5cf56db6bfa3da947c682d4d9270ec9e5"
Sep 05 03:17:28.748488 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:17:28.748460    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"068602bf3c4c781cda871806b7eef3a5cf56db6bfa3da947c682d4d9270ec9e5\": container with ID starting with 068602bf3c4c781cda871806b7eef3a5cf56db6bfa3da947c682d4d9270ec9e5 not found: ID does not exist" containerID="068602bf3c4c781cda871806b7eef3a5cf56db6bfa3da947c682d4d9270ec9e5"
Sep 05 03:17:28.748601 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.748494    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068602bf3c4c781cda871806b7eef3a5cf56db6bfa3da947c682d4d9270ec9e5"} err="failed to get container status \"068602bf3c4c781cda871806b7eef3a5cf56db6bfa3da947c682d4d9270ec9e5\": rpc error: code = NotFound desc = could not find container \"068602bf3c4c781cda871806b7eef3a5cf56db6bfa3da947c682d4d9270ec9e5\": container with ID starting with 068602bf3c4c781cda871806b7eef3a5cf56db6bfa3da947c682d4d9270ec9e5 not found: ID does not exist"
Sep 05 03:17:28.753705 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.753675    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"]
Sep 05 03:17:28.757891 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.757864    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-8xcct"]
Sep 05 03:17:28.991814 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:28.991729    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1d17008-b031-4753-8f92-05205a11fc61" path="/var/lib/kubelet/pods/c1d17008-b031-4753-8f92-05205a11fc61/volumes"
Sep 05 03:17:31.334048 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:31.334011    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 05 03:17:40.918035 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.917990    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"]
Sep 05 03:17:40.918613 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.918591    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main"
Sep 05 03:17:40.918707 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.918615    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main"
Sep 05 03:17:40.918707 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.918643    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="storage-initializer"
Sep 05 03:17:40.918707 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.918653    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="storage-initializer"
Sep 05 03:17:40.918873 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.918754    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1d17008-b031-4753-8f92-05205a11fc61" containerName="main"
Sep 05 03:17:40.922139 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.922113    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:40.924423 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.924392    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 05 03:17:40.924582 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.924481    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 05 03:17:40.924582 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.924568    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:17:40.924831 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.924806    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 05 03:17:40.924964 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.924940    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-rpdns\""
Sep 05 03:17:40.925060 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.925005    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 05 03:17:40.936527 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.936503    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"]
Sep 05 03:17:40.946814 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:40.946771    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:17:41.107070 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.107021    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldwvw\" (UniqueName: \"kubernetes.io/projected/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-kube-api-access-ldwvw\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.107275 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.107109    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-home\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.107275 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.107139    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-kserve-provision-location\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.107275 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.107177    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-model-cache\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.107275 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.107223    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-tmp-dir\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.107275 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.107251    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-dshm\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.107275 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.107268    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-tls-certs\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.107609 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.107320    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-cabundle-cert\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.172376 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.172287    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:17:41.207951 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.207914    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-dshm\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.207951 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.207954    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-tls-certs\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.208212 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.207984    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-cabundle-cert\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.208212 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.208056    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwvw\" (UniqueName: \"kubernetes.io/projected/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-kube-api-access-ldwvw\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.208212 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.208122    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-home\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.208212 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.208159    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-kserve-provision-location\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.208212 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.208190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-model-cache\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.208519 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.208234    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-tmp-dir\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.208644 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.208618    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-tmp-dir\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.208842 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.208816    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-kserve-provision-location\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.208926 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.208910    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-home\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.209047 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.209013    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-model-cache\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.209231 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.209209    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-cabundle-cert\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.210528 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.210500    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-dshm\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.210762 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.210741    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-tls-certs\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.215993 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.215968    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwvw\" (UniqueName: \"kubernetes.io/projected/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-kube-api-access-ldwvw\") pod \"stop-feature-test-kserve-6cc696dc5d-l9fbd\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.232841 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.232802    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:41.333186 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.333144    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 05 03:17:41.520254 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.520167    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"]
Sep 05 03:17:41.523316 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:17:41.523287    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c1a2c8_3207_4d7b_8064_ee2f12fe1eaf.slice/crio-205edbb8bdef0730f9cabab84a1a470a08748cac9fb544c6aca9f19de940a68c WatchSource:0}: Error finding container 205edbb8bdef0730f9cabab84a1a470a08748cac9fb544c6aca9f19de940a68c: Status 404 returned error can't find the container with id 205edbb8bdef0730f9cabab84a1a470a08748cac9fb544c6aca9f19de940a68c
Sep 05 03:17:41.805181 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.805087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" event={"ID":"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf","Type":"ContainerStarted","Data":"9a56e553b6941d5f9c722289e876edbdee4364bc6ae3a7ee5fa55e3dc5a9abe0"}
Sep 05 03:17:41.805181 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:41.805147    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" event={"ID":"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf","Type":"ContainerStarted","Data":"205edbb8bdef0730f9cabab84a1a470a08748cac9fb544c6aca9f19de940a68c"}
Sep 05 03:17:44.829136 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:44.829101    2557 generic.go:356] "Generic (PLEG): container finished" podID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerID="9a56e553b6941d5f9c722289e876edbdee4364bc6ae3a7ee5fa55e3dc5a9abe0" exitCode=0
Sep 05 03:17:44.829612 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:44.829179    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" event={"ID":"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf","Type":"ContainerDied","Data":"9a56e553b6941d5f9c722289e876edbdee4364bc6ae3a7ee5fa55e3dc5a9abe0"}
Sep 05 03:17:45.838820 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:45.838783    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" event={"ID":"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf","Type":"ContainerStarted","Data":"f88ac9dd0d55d40fbb860cea8eb2415102f900ad977c21e649fb39e302361352"}
Sep 05 03:17:45.862238 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:45.862165    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" podStartSLOduration=3.443076142 podStartE2EDuration="5.862145949s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.419069807s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:17:45.858974207 +0000 UTC m=+3569.414130530" watchObservedRunningTime="2026-09-05 03:17:45.862145949 +0000 UTC m=+3569.417302257"
Sep 05 03:17:46.233667 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:46.233565    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:50.007052 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:50.007005    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:17:51.233500 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:51.233447    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:17:51.234908 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:51.234874    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Sep 05 03:17:51.333923 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:17:51.333879    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 05 03:18:01.233849 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:18:01.233801    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Sep 05 03:18:01.333393 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:18:01.333349    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 05 03:18:11.233345 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:18:11.233245    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Sep 05 03:18:11.333620 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:18:11.333573    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 05 03:18:17.291535 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:18:17.291501    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:18:17.303646 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:18:17.303618    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:18:21.233793 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:18:21.233738    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Sep 05 03:18:21.333791 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:18:21.333748    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 05 03:18:31.234044 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:18:31.233986    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Sep 05 03:18:31.333549 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:18:31.333505    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 05 03:18:41.233383 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:18:41.233335    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Sep 05 03:18:41.333569 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:18:41.333522    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 05 03:18:51.233483 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:18:51.233410    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Sep 05 03:18:51.333336 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:18:51.333280    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 05 03:19:01.233786 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:01.233733    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Sep 05 03:19:01.343776 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:01.343741    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:19:01.355954 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:01.355924    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:19:09.226699 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:09.226658    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"]
Sep 05 03:19:09.228998 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:09.228947    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:19:09.292544 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:09.292495    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:19:11.233496 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:11.233417    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Sep 05 03:19:21.233788 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:21.233739    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Sep 05 03:19:24.254688 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:24.254647    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main" containerID="cri-o://0f2c99fe39a7d752195b93c2949dfb101084f19f2b07ebe858436fdd424a63d7" gracePeriod=45
Sep 05 03:19:24.345090 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:24.345036    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 05 03:19:25.310023 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.309992    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:19:25.435498 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.435456    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-dshm\") pod \"fce67741-fb12-4f40-b8e0-0ccb8931141a\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") "
Sep 05 03:19:25.435710 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.435508    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-tmp-dir\") pod \"fce67741-fb12-4f40-b8e0-0ccb8931141a\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") "
Sep 05 03:19:25.435710 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.435536    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-kserve-provision-location\") pod \"fce67741-fb12-4f40-b8e0-0ccb8931141a\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") "
Sep 05 03:19:25.435710 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.435579    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-model-cache\") pod \"fce67741-fb12-4f40-b8e0-0ccb8931141a\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") "
Sep 05 03:19:25.435710 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.435648    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce67741-fb12-4f40-b8e0-0ccb8931141a-tls-certs\") pod \"fce67741-fb12-4f40-b8e0-0ccb8931141a\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") "
Sep 05 03:19:25.435710 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.435673    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-home\") pod \"fce67741-fb12-4f40-b8e0-0ccb8931141a\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") "
Sep 05 03:19:25.435710 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.435700    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gf77\" (UniqueName: \"kubernetes.io/projected/fce67741-fb12-4f40-b8e0-0ccb8931141a-kube-api-access-2gf77\") pod \"fce67741-fb12-4f40-b8e0-0ccb8931141a\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") "
Sep 05 03:19:25.436051 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.435731    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce67741-fb12-4f40-b8e0-0ccb8931141a-cabundle-cert\") pod \"fce67741-fb12-4f40-b8e0-0ccb8931141a\" (UID: \"fce67741-fb12-4f40-b8e0-0ccb8931141a\") "
Sep 05 03:19:25.436051 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.435945    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-model-cache" (OuterVolumeSpecName: "model-cache") pod "fce67741-fb12-4f40-b8e0-0ccb8931141a" (UID: "fce67741-fb12-4f40-b8e0-0ccb8931141a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:19:25.436164 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.436094    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:19:25.436508 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.436467    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce67741-fb12-4f40-b8e0-0ccb8931141a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fce67741-fb12-4f40-b8e0-0ccb8931141a" (UID: "fce67741-fb12-4f40-b8e0-0ccb8931141a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:19:25.436762 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.436707    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-home" (OuterVolumeSpecName: "home") pod "fce67741-fb12-4f40-b8e0-0ccb8931141a" (UID: "fce67741-fb12-4f40-b8e0-0ccb8931141a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:19:25.438235 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.438200    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-dshm" (OuterVolumeSpecName: "dshm") pod "fce67741-fb12-4f40-b8e0-0ccb8931141a" (UID: "fce67741-fb12-4f40-b8e0-0ccb8931141a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:19:25.438499 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.438471    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce67741-fb12-4f40-b8e0-0ccb8931141a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fce67741-fb12-4f40-b8e0-0ccb8931141a" (UID: "fce67741-fb12-4f40-b8e0-0ccb8931141a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:19:25.438578 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.438518    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce67741-fb12-4f40-b8e0-0ccb8931141a-kube-api-access-2gf77" (OuterVolumeSpecName: "kube-api-access-2gf77") pod "fce67741-fb12-4f40-b8e0-0ccb8931141a" (UID: "fce67741-fb12-4f40-b8e0-0ccb8931141a"). InnerVolumeSpecName "kube-api-access-2gf77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:19:25.442295 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.442273    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fce67741-fb12-4f40-b8e0-0ccb8931141a" (UID: "fce67741-fb12-4f40-b8e0-0ccb8931141a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:19:25.453171 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.453138    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fce67741-fb12-4f40-b8e0-0ccb8931141a" (UID: "fce67741-fb12-4f40-b8e0-0ccb8931141a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:19:25.492078 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.492039    2557 generic.go:356] "Generic (PLEG): container finished" podID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerID="0f2c99fe39a7d752195b93c2949dfb101084f19f2b07ebe858436fdd424a63d7" exitCode=0
Sep 05 03:19:25.492251 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.492096    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"
Sep 05 03:19:25.492251 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.492131    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" event={"ID":"fce67741-fb12-4f40-b8e0-0ccb8931141a","Type":"ContainerDied","Data":"0f2c99fe39a7d752195b93c2949dfb101084f19f2b07ebe858436fdd424a63d7"}
Sep 05 03:19:25.492251 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.492207    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8" event={"ID":"fce67741-fb12-4f40-b8e0-0ccb8931141a","Type":"ContainerDied","Data":"da6ffe51c6eee3ccf2db2b2592e2eec4164b64a7666f6840dec9d7bea0f59ebd"}
Sep 05 03:19:25.492251 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.492246    2557 scope.go:117] "RemoveContainer" containerID="0f2c99fe39a7d752195b93c2949dfb101084f19f2b07ebe858436fdd424a63d7"
Sep 05 03:19:25.504208 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.504164    2557 scope.go:117] "RemoveContainer" containerID="40f35e6ef11f15424733f9dba4759b31ada9596e689d2a6984a4cd2dad1c89cb"
Sep 05 03:19:25.514351 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.514332    2557 scope.go:117] "RemoveContainer" containerID="0f2c99fe39a7d752195b93c2949dfb101084f19f2b07ebe858436fdd424a63d7"
Sep 05 03:19:25.514700 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:19:25.514677    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2c99fe39a7d752195b93c2949dfb101084f19f2b07ebe858436fdd424a63d7\": container with ID starting with 0f2c99fe39a7d752195b93c2949dfb101084f19f2b07ebe858436fdd424a63d7 not found: ID does not exist" containerID="0f2c99fe39a7d752195b93c2949dfb101084f19f2b07ebe858436fdd424a63d7"
Sep 05 03:19:25.514794 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.514713    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2c99fe39a7d752195b93c2949dfb101084f19f2b07ebe858436fdd424a63d7"} err="failed to get container status \"0f2c99fe39a7d752195b93c2949dfb101084f19f2b07ebe858436fdd424a63d7\": rpc error: code = NotFound desc = could not find container \"0f2c99fe39a7d752195b93c2949dfb101084f19f2b07ebe858436fdd424a63d7\": container with ID starting with 0f2c99fe39a7d752195b93c2949dfb101084f19f2b07ebe858436fdd424a63d7 not found: ID does not exist"
Sep 05 03:19:25.514794 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.514737    2557 scope.go:117] "RemoveContainer" containerID="40f35e6ef11f15424733f9dba4759b31ada9596e689d2a6984a4cd2dad1c89cb"
Sep 05 03:19:25.515058 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:19:25.515039    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f35e6ef11f15424733f9dba4759b31ada9596e689d2a6984a4cd2dad1c89cb\": container with ID starting with 40f35e6ef11f15424733f9dba4759b31ada9596e689d2a6984a4cd2dad1c89cb not found: ID does not exist" containerID="40f35e6ef11f15424733f9dba4759b31ada9596e689d2a6984a4cd2dad1c89cb"
Sep 05 03:19:25.515116 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.515067    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f35e6ef11f15424733f9dba4759b31ada9596e689d2a6984a4cd2dad1c89cb"} err="failed to get container status \"40f35e6ef11f15424733f9dba4759b31ada9596e689d2a6984a4cd2dad1c89cb\": rpc error: code = NotFound desc = could not find container \"40f35e6ef11f15424733f9dba4759b31ada9596e689d2a6984a4cd2dad1c89cb\": container with ID starting with 40f35e6ef11f15424733f9dba4759b31ada9596e689d2a6984a4cd2dad1c89cb not found: ID does not exist"
Sep 05 03:19:25.515658 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.515638    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"]
Sep 05 03:19:25.520711 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.520685    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79c695dc8c-rf8m8"]
Sep 05 03:19:25.537141 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.537114    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce67741-fb12-4f40-b8e0-0ccb8931141a-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:19:25.537141 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.537141    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:19:25.537262 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.537151    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gf77\" (UniqueName: \"kubernetes.io/projected/fce67741-fb12-4f40-b8e0-0ccb8931141a-kube-api-access-2gf77\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:19:25.537262 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.537163    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce67741-fb12-4f40-b8e0-0ccb8931141a-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:19:25.537262 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.537172    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:19:25.537262 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.537180    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:19:25.537262 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:25.537188    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce67741-fb12-4f40-b8e0-0ccb8931141a-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:19:26.993894 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:26.993846    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" path="/var/lib/kubelet/pods/fce67741-fb12-4f40-b8e0-0ccb8931141a/volumes"
Sep 05 03:19:31.233674 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:31.233634    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Sep 05 03:19:41.234280 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:41.234168    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Sep 05 03:19:51.243866 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:51.243828    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:19:51.251829 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:51.251799    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:19:53.325104 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:53.325062    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"]
Sep 05 03:19:53.325914 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:53.325760    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:19:53.421455 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:53.421407    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:19:59.258022 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:59.257986    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:19:59.353672 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:19:59.353641    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:04.212541 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:04.212495    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"]
Sep 05 03:20:04.212943 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:04.212700    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2" podUID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerName="istio-proxy" containerID="cri-o://81a3411fb5cdae972e1ad8069371b4b1033380c1af49e70c9cb4e84407fa7ed7" gracePeriod=30
Sep 05 03:20:04.259182 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:04.259129    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:07.264410 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.264371    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"]
Sep 05 03:20:07.264786 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.264774    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="storage-initializer"
Sep 05 03:20:07.264833 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.264790    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="storage-initializer"
Sep 05 03:20:07.264833 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.264807    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main"
Sep 05 03:20:07.264833 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.264813    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main"
Sep 05 03:20:07.264931 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.264885    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce67741-fb12-4f40-b8e0-0ccb8931141a" containerName="main"
Sep 05 03:20:07.268010 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.267984    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.275193 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.275153    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-dtt9d\""
Sep 05 03:20:07.275312 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.275205    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 05 03:20:07.275312 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.275209    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 05 03:20:07.275312 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.275288    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:20:07.275519 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.275378    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 05 03:20:07.275813 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.275795    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 05 03:20:07.275863 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.275795    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-99xqq\""
Sep 05 03:20:07.282739 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.282718    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"]
Sep 05 03:20:07.306086 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.306065    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:07.317317 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.317286    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.317420 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.317329    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.317420 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.317353    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.317420 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.317373    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.317420 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.317388    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de020384-2f15-42b1-9e1f-3792dbe10578-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.317420 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.317406    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhsn2\" (UniqueName: \"kubernetes.io/projected/de020384-2f15-42b1-9e1f-3792dbe10578-kube-api-access-vhsn2\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.317659 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.317505    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.317659 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.317542    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de020384-2f15-42b1-9e1f-3792dbe10578-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.418377 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.418343    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.418578 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.418385    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.418578 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.418404    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.418578 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.418431    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.418578 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.418480    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de020384-2f15-42b1-9e1f-3792dbe10578-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.418578 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.418505    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhsn2\" (UniqueName: \"kubernetes.io/projected/de020384-2f15-42b1-9e1f-3792dbe10578-kube-api-access-vhsn2\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.418578 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.418538    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.418932 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.418593    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de020384-2f15-42b1-9e1f-3792dbe10578-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.418932 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.418848    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.418932 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.418897    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.419053 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.418937    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.419053 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.418949    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.419329 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.419312    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de020384-2f15-42b1-9e1f-3792dbe10578-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.420899 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.420878    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.420997 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.420981    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de020384-2f15-42b1-9e1f-3792dbe10578-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.426023 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.426001    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhsn2\" (UniqueName: \"kubernetes.io/projected/de020384-2f15-42b1-9e1f-3792dbe10578-kube-api-access-vhsn2\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.511236 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.511180    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:07.578184 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.578153    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:07.868865 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:07.868829    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"]
Sep 05 03:20:07.870087 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:20:07.870053    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde020384_2f15_42b1_9e1f_3792dbe10578.slice/crio-9c8ede2d823d1e6252a3566c6c52af158c63316f3f511af4b63f963aa07f6372 WatchSource:0}: Error finding container 9c8ede2d823d1e6252a3566c6c52af158c63316f3f511af4b63f963aa07f6372: Status 404 returned error can't find the container with id 9c8ede2d823d1e6252a3566c6c52af158c63316f3f511af4b63f963aa07f6372
Sep 05 03:20:08.354774 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:08.354735    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main" containerID="cri-o://f88ac9dd0d55d40fbb860cea8eb2415102f900ad977c21e649fb39e302361352" gracePeriod=45
Sep 05 03:20:08.737256 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:08.737147    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" event={"ID":"de020384-2f15-42b1-9e1f-3792dbe10578","Type":"ContainerStarted","Data":"70d1ba9d342cd641a8b4e7c15b54372ae4b2c0720cb09271e5216947e4bf012a"}
Sep 05 03:20:08.737256 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:08.737245    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" event={"ID":"de020384-2f15-42b1-9e1f-3792dbe10578","Type":"ContainerStarted","Data":"9c8ede2d823d1e6252a3566c6c52af158c63316f3f511af4b63f963aa07f6372"}
Sep 05 03:20:08.737557 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:08.737385    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:09.118005 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.117974    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:20:09.237884 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.237858    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-dshm\") pod \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") "
Sep 05 03:20:09.238024 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.237902    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-tmp-dir\") pod \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") "
Sep 05 03:20:09.238024 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.237946    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-model-cache\") pod \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") "
Sep 05 03:20:09.238024 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.237974    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-kserve-provision-location\") pod \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") "
Sep 05 03:20:09.238024 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.238013    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-tls-certs\") pod \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") "
Sep 05 03:20:09.238236 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.238075    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldwvw\" (UniqueName: \"kubernetes.io/projected/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-kube-api-access-ldwvw\") pod \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") "
Sep 05 03:20:09.238236 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.238105    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-home\") pod \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") "
Sep 05 03:20:09.238236 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.238143    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-cabundle-cert\") pod \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\" (UID: \"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf\") "
Sep 05 03:20:09.239002 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.238790    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" (UID: "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:20:09.239002 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.238925    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" (UID: "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:20:09.239709 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.239676    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-home" (OuterVolumeSpecName: "home") pod "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" (UID: "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:20:09.241404 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.241380    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-dshm" (OuterVolumeSpecName: "dshm") pod "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" (UID: "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:20:09.242302 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.241780    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" (UID: "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:20:09.242939 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.242910    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-kube-api-access-ldwvw" (OuterVolumeSpecName: "kube-api-access-ldwvw") pod "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" (UID: "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf"). InnerVolumeSpecName "kube-api-access-ldwvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:20:09.247287 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.247262    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" (UID: "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:20:09.253447 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.253409    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" (UID: "e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:20:09.339586 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.339554    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.339586 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.339584    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.339803 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.339594    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.339803 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.339603    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldwvw\" (UniqueName: \"kubernetes.io/projected/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-kube-api-access-ldwvw\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.339803 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.339611    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.339803 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.339625    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.339803 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.339636    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.339803 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.339643    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.487213 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.487185    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:20:09.540710 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.540661    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-workload-socket\") pod \"61b9d643-fbbf-43c6-940d-621a39c648b6\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") "
Sep 05 03:20:09.540881 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.540739    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-envoy\") pod \"61b9d643-fbbf-43c6-940d-621a39c648b6\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") "
Sep 05 03:20:09.540881 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.540788    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsjpj\" (UniqueName: \"kubernetes.io/projected/61b9d643-fbbf-43c6-940d-621a39c648b6-kube-api-access-fsjpj\") pod \"61b9d643-fbbf-43c6-940d-621a39c648b6\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") "
Sep 05 03:20:09.540881 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.540832    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-podinfo\") pod \"61b9d643-fbbf-43c6-940d-621a39c648b6\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") "
Sep 05 03:20:09.541059 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.540887    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-data\") pod \"61b9d643-fbbf-43c6-940d-621a39c648b6\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") "
Sep 05 03:20:09.541059 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.540914    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "61b9d643-fbbf-43c6-940d-621a39c648b6" (UID: "61b9d643-fbbf-43c6-940d-621a39c648b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:20:09.541059 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.540935    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-credential-socket\") pod \"61b9d643-fbbf-43c6-940d-621a39c648b6\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") "
Sep 05 03:20:09.541059 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.540966    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-token\") pod \"61b9d643-fbbf-43c6-940d-621a39c648b6\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") "
Sep 05 03:20:09.541059 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.540993    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b9d643-fbbf-43c6-940d-621a39c648b6-istiod-ca-cert\") pod \"61b9d643-fbbf-43c6-940d-621a39c648b6\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") "
Sep 05 03:20:09.541059 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.541037    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-workload-certs\") pod \"61b9d643-fbbf-43c6-940d-621a39c648b6\" (UID: \"61b9d643-fbbf-43c6-940d-621a39c648b6\") "
Sep 05 03:20:09.541389 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.541336    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-workload-socket\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.541772 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.541608    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "61b9d643-fbbf-43c6-940d-621a39c648b6" (UID: "61b9d643-fbbf-43c6-940d-621a39c648b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:20:09.541772 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.541733    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "61b9d643-fbbf-43c6-940d-621a39c648b6" (UID: "61b9d643-fbbf-43c6-940d-621a39c648b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:20:09.542026 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.541898    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "61b9d643-fbbf-43c6-940d-621a39c648b6" (UID: "61b9d643-fbbf-43c6-940d-621a39c648b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:20:09.542633 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.542274    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b9d643-fbbf-43c6-940d-621a39c648b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "61b9d643-fbbf-43c6-940d-621a39c648b6" (UID: "61b9d643-fbbf-43c6-940d-621a39c648b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:20:09.544354 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.544240    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "61b9d643-fbbf-43c6-940d-621a39c648b6" (UID: "61b9d643-fbbf-43c6-940d-621a39c648b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:20:09.544354 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.544361    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "61b9d643-fbbf-43c6-940d-621a39c648b6" (UID: "61b9d643-fbbf-43c6-940d-621a39c648b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:20:09.544354 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.544942    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b9d643-fbbf-43c6-940d-621a39c648b6-kube-api-access-fsjpj" (OuterVolumeSpecName: "kube-api-access-fsjpj") pod "61b9d643-fbbf-43c6-940d-621a39c648b6" (UID: "61b9d643-fbbf-43c6-940d-621a39c648b6"). InnerVolumeSpecName "kube-api-access-fsjpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:20:09.546530 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.546500    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "61b9d643-fbbf-43c6-940d-621a39c648b6" (UID: "61b9d643-fbbf-43c6-940d-621a39c648b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 03:20:09.642883 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.642835    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-envoy\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.642883 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.642881    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsjpj\" (UniqueName: \"kubernetes.io/projected/61b9d643-fbbf-43c6-940d-621a39c648b6-kube-api-access-fsjpj\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.643160 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.642897    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-podinfo\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.643160 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.642911    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-data\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.643160 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.642925    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-credential-socket\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.643160 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.642937    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b9d643-fbbf-43c6-940d-621a39c648b6-istio-token\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.643160 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.642950    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b9d643-fbbf-43c6-940d-621a39c648b6-istiod-ca-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.643160 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.642964    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b9d643-fbbf-43c6-940d-621a39c648b6-workload-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:20:09.744921 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.744877    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" event={"ID":"de020384-2f15-42b1-9e1f-3792dbe10578","Type":"ContainerStarted","Data":"1529ef563eca81492e70f50d6e9d8265fea5207e15f73d32e764d2a71e1ce3b4"}
Sep 05 03:20:09.746698 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.746662    2557 generic.go:356] "Generic (PLEG): container finished" podID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerID="f88ac9dd0d55d40fbb860cea8eb2415102f900ad977c21e649fb39e302361352" exitCode=0
Sep 05 03:20:09.746828 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.746703    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"
Sep 05 03:20:09.746828 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.746748    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" event={"ID":"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf","Type":"ContainerDied","Data":"f88ac9dd0d55d40fbb860cea8eb2415102f900ad977c21e649fb39e302361352"}
Sep 05 03:20:09.746828 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.746806    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd" event={"ID":"e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf","Type":"ContainerDied","Data":"205edbb8bdef0730f9cabab84a1a470a08748cac9fb544c6aca9f19de940a68c"}
Sep 05 03:20:09.747006 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.746838    2557 scope.go:117] "RemoveContainer" containerID="f88ac9dd0d55d40fbb860cea8eb2415102f900ad977c21e649fb39e302361352"
Sep 05 03:20:09.749063 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.749034    2557 generic.go:356] "Generic (PLEG): container finished" podID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerID="81a3411fb5cdae972e1ad8069371b4b1033380c1af49e70c9cb4e84407fa7ed7" exitCode=0
Sep 05 03:20:09.749155 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.749067    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2" event={"ID":"61b9d643-fbbf-43c6-940d-621a39c648b6","Type":"ContainerDied","Data":"81a3411fb5cdae972e1ad8069371b4b1033380c1af49e70c9cb4e84407fa7ed7"}
Sep 05 03:20:09.749155 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.749138    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2" event={"ID":"61b9d643-fbbf-43c6-940d-621a39c648b6","Type":"ContainerDied","Data":"97b1f1a68a532a774fac0b9397a527f81284cf27c1f5d25d6dad00925679ec52"}
Sep 05 03:20:09.749288 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.749076    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"
Sep 05 03:20:09.760968 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.760841    2557 scope.go:117] "RemoveContainer" containerID="9a56e553b6941d5f9c722289e876edbdee4364bc6ae3a7ee5fa55e3dc5a9abe0"
Sep 05 03:20:09.781361 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.779870    2557 scope.go:117] "RemoveContainer" containerID="f88ac9dd0d55d40fbb860cea8eb2415102f900ad977c21e649fb39e302361352"
Sep 05 03:20:09.784417 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:20:09.784202    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88ac9dd0d55d40fbb860cea8eb2415102f900ad977c21e649fb39e302361352\": container with ID starting with f88ac9dd0d55d40fbb860cea8eb2415102f900ad977c21e649fb39e302361352 not found: ID does not exist" containerID="f88ac9dd0d55d40fbb860cea8eb2415102f900ad977c21e649fb39e302361352"
Sep 05 03:20:09.784417 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.784263    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88ac9dd0d55d40fbb860cea8eb2415102f900ad977c21e649fb39e302361352"} err="failed to get container status \"f88ac9dd0d55d40fbb860cea8eb2415102f900ad977c21e649fb39e302361352\": rpc error: code = NotFound desc = could not find container \"f88ac9dd0d55d40fbb860cea8eb2415102f900ad977c21e649fb39e302361352\": container with ID starting with f88ac9dd0d55d40fbb860cea8eb2415102f900ad977c21e649fb39e302361352 not found: ID does not exist"
Sep 05 03:20:09.784417 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.784294    2557 scope.go:117] "RemoveContainer" containerID="9a56e553b6941d5f9c722289e876edbdee4364bc6ae3a7ee5fa55e3dc5a9abe0"
Sep 05 03:20:09.790575 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:20:09.790506    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a56e553b6941d5f9c722289e876edbdee4364bc6ae3a7ee5fa55e3dc5a9abe0\": container with ID starting with 9a56e553b6941d5f9c722289e876edbdee4364bc6ae3a7ee5fa55e3dc5a9abe0 not found: ID does not exist" containerID="9a56e553b6941d5f9c722289e876edbdee4364bc6ae3a7ee5fa55e3dc5a9abe0"
Sep 05 03:20:09.790575 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.790562    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a56e553b6941d5f9c722289e876edbdee4364bc6ae3a7ee5fa55e3dc5a9abe0"} err="failed to get container status \"9a56e553b6941d5f9c722289e876edbdee4364bc6ae3a7ee5fa55e3dc5a9abe0\": rpc error: code = NotFound desc = could not find container \"9a56e553b6941d5f9c722289e876edbdee4364bc6ae3a7ee5fa55e3dc5a9abe0\": container with ID starting with 9a56e553b6941d5f9c722289e876edbdee4364bc6ae3a7ee5fa55e3dc5a9abe0 not found: ID does not exist"
Sep 05 03:20:09.790835 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.790588    2557 scope.go:117] "RemoveContainer" containerID="81a3411fb5cdae972e1ad8069371b4b1033380c1af49e70c9cb4e84407fa7ed7"
Sep 05 03:20:09.802005 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.801975    2557 scope.go:117] "RemoveContainer" containerID="a59a4a621df7085087f4bb2f10419755edac7c675a6a3286425b3b4ef723a7d3"
Sep 05 03:20:09.809482 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.809425    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"]
Sep 05 03:20:09.813265 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.813247    2557 scope.go:117] "RemoveContainer" containerID="81a3411fb5cdae972e1ad8069371b4b1033380c1af49e70c9cb4e84407fa7ed7"
Sep 05 03:20:09.813618 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:20:09.813592    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a3411fb5cdae972e1ad8069371b4b1033380c1af49e70c9cb4e84407fa7ed7\": container with ID starting with 81a3411fb5cdae972e1ad8069371b4b1033380c1af49e70c9cb4e84407fa7ed7 not found: ID does not exist" containerID="81a3411fb5cdae972e1ad8069371b4b1033380c1af49e70c9cb4e84407fa7ed7"
Sep 05 03:20:09.813677 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.813627    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a3411fb5cdae972e1ad8069371b4b1033380c1af49e70c9cb4e84407fa7ed7"} err="failed to get container status \"81a3411fb5cdae972e1ad8069371b4b1033380c1af49e70c9cb4e84407fa7ed7\": rpc error: code = NotFound desc = could not find container \"81a3411fb5cdae972e1ad8069371b4b1033380c1af49e70c9cb4e84407fa7ed7\": container with ID starting with 81a3411fb5cdae972e1ad8069371b4b1033380c1af49e70c9cb4e84407fa7ed7 not found: ID does not exist"
Sep 05 03:20:09.813677 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.813646    2557 scope.go:117] "RemoveContainer" containerID="a59a4a621df7085087f4bb2f10419755edac7c675a6a3286425b3b4ef723a7d3"
Sep 05 03:20:09.813932 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:20:09.813912    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59a4a621df7085087f4bb2f10419755edac7c675a6a3286425b3b4ef723a7d3\": container with ID starting with a59a4a621df7085087f4bb2f10419755edac7c675a6a3286425b3b4ef723a7d3 not found: ID does not exist" containerID="a59a4a621df7085087f4bb2f10419755edac7c675a6a3286425b3b4ef723a7d3"
Sep 05 03:20:09.813977 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.813948    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59a4a621df7085087f4bb2f10419755edac7c675a6a3286425b3b4ef723a7d3"} err="failed to get container status \"a59a4a621df7085087f4bb2f10419755edac7c675a6a3286425b3b4ef723a7d3\": rpc error: code = NotFound desc = could not find container \"a59a4a621df7085087f4bb2f10419755edac7c675a6a3286425b3b4ef723a7d3\": container with ID starting with a59a4a621df7085087f4bb2f10419755edac7c675a6a3286425b3b4ef723a7d3 not found: ID does not exist"
Sep 05 03:20:09.820980 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.820956    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-8r6r2"]
Sep 05 03:20:09.835049 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.834993    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"]
Sep 05 03:20:09.838690 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:09.838653    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6cc696dc5d-l9fbd"]
Sep 05 03:20:10.993342 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:10.993307    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61b9d643-fbbf-43c6-940d-621a39c648b6" path="/var/lib/kubelet/pods/61b9d643-fbbf-43c6-940d-621a39c648b6/volumes"
Sep 05 03:20:10.993959 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:10.993934    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" path="/var/lib/kubelet/pods/e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf/volumes"
Sep 05 03:20:11.765224 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:11.765186    2557 generic.go:356] "Generic (PLEG): container finished" podID="de020384-2f15-42b1-9e1f-3792dbe10578" containerID="1529ef563eca81492e70f50d6e9d8265fea5207e15f73d32e764d2a71e1ce3b4" exitCode=0
Sep 05 03:20:11.765382 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:11.765262    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" event={"ID":"de020384-2f15-42b1-9e1f-3792dbe10578","Type":"ContainerDied","Data":"1529ef563eca81492e70f50d6e9d8265fea5207e15f73d32e764d2a71e1ce3b4"}
Sep 05 03:20:12.773594 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:12.773553    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" event={"ID":"de020384-2f15-42b1-9e1f-3792dbe10578","Type":"ContainerStarted","Data":"1b86e5870ff1095bfa9fd1a472ab3be2f48f9775a16a2a60eed14d0cc101585b"}
Sep 05 03:20:12.798130 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:12.798063    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podStartSLOduration=3.341067951 podStartE2EDuration="5.798044401s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.45697645s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:20:12.795300852 +0000 UTC m=+3716.350457161" watchObservedRunningTime="2026-09-05 03:20:12.798044401 +0000 UTC m=+3716.353200710"
Sep 05 03:20:17.578717 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:17.578672    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:17.579157 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:17.578750    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:17.580238 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:17.580199    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8001/health\": dial tcp 10.133.0.99:8001: connect: connection refused"
Sep 05 03:20:17.607824 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:17.607786    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:20:21.279476 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:21.279412    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:26.409194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:26.409153    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:26.739799 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:26.739698    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:27.579187 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:27.579121    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8001/health\": dial tcp 10.133.0.99:8001: connect: connection refused"
Sep 05 03:20:32.302629 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:32.302581    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:37.579552 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:37.579495    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8001/health\": dial tcp 10.133.0.99:8001: connect: connection refused"
Sep 05 03:20:42.322369 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:42.322326    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:42.421676 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:42.421629    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:42.524469 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:42.524414    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:42.622643 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:42.622535    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:42.856004 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:42.855954    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:46.436002 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:46.435965    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:46.536802 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:46.536765    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:47.578982 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:47.578921    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8001/health\": dial tcp 10.133.0.99:8001: connect: connection refused"
Sep 05 03:20:48.447854 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:48.447809    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:49.390993 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.390947    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"]
Sep 05 03:20:49.391530 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.391507    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="storage-initializer"
Sep 05 03:20:49.391625 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.391532    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="storage-initializer"
Sep 05 03:20:49.391625 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.391544    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main"
Sep 05 03:20:49.391625 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.391552    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main"
Sep 05 03:20:49.391625 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.391568    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerName="istio-proxy"
Sep 05 03:20:49.391625 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.391577    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerName="istio-proxy"
Sep 05 03:20:49.391881 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.391688    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerName="istio-proxy"
Sep 05 03:20:49.391881 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.391702    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2c1a2c8-3207-4d7b-8064-ee2f12fe1eaf" containerName="main"
Sep 05 03:20:49.391881 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.391714    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerName="istio-proxy"
Sep 05 03:20:49.391881 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.391822    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerName="istio-proxy"
Sep 05 03:20:49.391881 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.391833    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="61b9d643-fbbf-43c6-940d-621a39c648b6" containerName="istio-proxy"
Sep 05 03:20:49.395355 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.395331    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.398103 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.398077    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 05 03:20:49.398224 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.398079    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 05 03:20:49.398224 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.398172    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-wbnrm\""
Sep 05 03:20:49.398408 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.398386    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 05 03:20:49.398610 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.398583    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:20:49.398960 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.398938    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 05 03:20:49.405800 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.405777    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"]
Sep 05 03:20:49.419144 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.419114    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:49.500307 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.500265    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxrxs\" (UniqueName: \"kubernetes.io/projected/819bf449-908c-47ac-85d5-917db6e76ee1-kube-api-access-cxrxs\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.500493 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.500352    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.500493 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.500388    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-dshm\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.500493 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.500476    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-model-cache\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.500611 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.500516    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-home\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.500611 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.500545    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819bf449-908c-47ac-85d5-917db6e76ee1-tls-certs\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.500683 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.500619    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-tmp-dir\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.500683 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.500662    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/819bf449-908c-47ac-85d5-917db6e76ee1-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.601624 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.601583    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-model-cache\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.601812 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.601640    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-home\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.601812 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.601671    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819bf449-908c-47ac-85d5-917db6e76ee1-tls-certs\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.601812 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.601718    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-tmp-dir\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.601812 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.601741    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/819bf449-908c-47ac-85d5-917db6e76ee1-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.601812 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.601791    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrxs\" (UniqueName: \"kubernetes.io/projected/819bf449-908c-47ac-85d5-917db6e76ee1-kube-api-access-cxrxs\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.602073 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.601842    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.602073 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.601873    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-dshm\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.602184 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.602073    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-model-cache\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.602184 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.602126    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-home\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.602297 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.602188    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-tmp-dir\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.602404 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.602384    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.602634 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.602614    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/819bf449-908c-47ac-85d5-917db6e76ee1-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.604031 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.604010    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-dshm\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.604398 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.604375    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819bf449-908c-47ac-85d5-917db6e76ee1-tls-certs\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.612899 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.612875    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrxs\" (UniqueName: \"kubernetes.io/projected/819bf449-908c-47ac-85d5-917db6e76ee1-kube-api-access-cxrxs\") pod \"lora-single-adapter-test-kserve-7898b797f6-5twpx\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:49.708325 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:49.708234    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:50.001462 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:50.001369    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"]
Sep 05 03:20:50.003184 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:20:50.003145    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819bf449_908c_47ac_85d5_917db6e76ee1.slice/crio-e17521f19fa561d840406714177e0e746cdc7104713f53baeb343674968c03f5 WatchSource:0}: Error finding container e17521f19fa561d840406714177e0e746cdc7104713f53baeb343674968c03f5: Status 404 returned error can't find the container with id e17521f19fa561d840406714177e0e746cdc7104713f53baeb343674968c03f5
Sep 05 03:20:51.005081 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:51.005041    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" event={"ID":"819bf449-908c-47ac-85d5-917db6e76ee1","Type":"ContainerStarted","Data":"f5cb19fb04ebc67ecc04dde2261559a9996624afb17a706739c435c54423a6bc"}
Sep 05 03:20:51.005534 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:51.005093    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" event={"ID":"819bf449-908c-47ac-85d5-917db6e76ee1","Type":"ContainerStarted","Data":"e17521f19fa561d840406714177e0e746cdc7104713f53baeb343674968c03f5"}
Sep 05 03:20:54.026456 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:54.026393    2557 generic.go:356] "Generic (PLEG): container finished" podID="819bf449-908c-47ac-85d5-917db6e76ee1" containerID="f5cb19fb04ebc67ecc04dde2261559a9996624afb17a706739c435c54423a6bc" exitCode=0
Sep 05 03:20:54.026967 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:54.026479    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" event={"ID":"819bf449-908c-47ac-85d5-917db6e76ee1","Type":"ContainerDied","Data":"f5cb19fb04ebc67ecc04dde2261559a9996624afb17a706739c435c54423a6bc"}
Sep 05 03:20:54.213415 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:54.213367    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:54.299567 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:54.299459    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:20:55.034394 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:55.034352    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" event={"ID":"819bf449-908c-47ac-85d5-917db6e76ee1","Type":"ContainerStarted","Data":"a06ca1968677d260a6432d3478dd720d4d3120455cb44da6b484c871c862762d"}
Sep 05 03:20:55.060187 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:55.060129    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" podStartSLOduration=2.909714186 podStartE2EDuration="6.060109428s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.150395242s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:20:55.058906387 +0000 UTC m=+3758.614062697" watchObservedRunningTime="2026-09-05 03:20:55.060109428 +0000 UTC m=+3758.615265737"
Sep 05 03:20:55.708897 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:55.708849    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:57.578796 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:57.578729    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8001/health\": dial tcp 10.133.0.99:8001: connect: connection refused"
Sep 05 03:20:59.708388 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:59.708344    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:20:59.710481 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:20:59.710432    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Sep 05 03:21:07.579217 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:21:07.579155    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8001/health\": dial tcp 10.133.0.99:8001: connect: connection refused"
Sep 05 03:21:09.709184 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:21:09.709084    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Sep 05 03:21:17.579310 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:21:17.579256    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8001/health\": dial tcp 10.133.0.99:8001: connect: connection refused"
Sep 05 03:21:19.709601 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:21:19.709543    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Sep 05 03:21:27.579534 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:21:27.579469    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8001/health\": dial tcp 10.133.0.99:8001: connect: connection refused"
Sep 05 03:21:29.709393 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:21:29.709333    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Sep 05 03:21:37.579117 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:21:37.579062    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8001/health\": dial tcp 10.133.0.99:8001: connect: connection refused"
Sep 05 03:21:39.708728 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:21:39.708675    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Sep 05 03:21:47.579742 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:21:47.579686    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8001/health\": dial tcp 10.133.0.99:8001: connect: connection refused"
Sep 05 03:21:49.709008 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:21:49.708942    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Sep 05 03:21:57.579930 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:21:57.579383    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8001/health\": dial tcp 10.133.0.99:8001: connect: connection refused"
Sep 05 03:21:59.709111 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:21:59.709059    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Sep 05 03:22:07.578691 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:07.578603    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8001/health\": dial tcp 10.133.0.99:8001: connect: connection refused"
Sep 05 03:22:09.709149 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:09.709096    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Sep 05 03:22:15.848590 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:15.848549    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:22:16.929250 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:16.929181    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:22:17.591348 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:17.591310    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:22:17.610804 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:17.610766    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:22:19.718950 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:19.718913    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:22:19.726923 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:19.726892    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:22:30.792404 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:30.792361    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:22:30.796658 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:30.796619    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"]
Sep 05 03:22:30.882233 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:30.882183    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:22:45.821515 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:45.821388    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main" containerID="cri-o://1b86e5870ff1095bfa9fd1a472ab3be2f48f9775a16a2a60eed14d0cc101585b" gracePeriod=45
Sep 05 03:22:46.735208 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:46.735156    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="llm-d-routing-sidecar" containerID="cri-o://70d1ba9d342cd641a8b4e7c15b54372ae4b2c0720cb09271e5216947e4bf012a" gracePeriod=45
Sep 05 03:22:46.804303 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:46.804263    2557 generic.go:356] "Generic (PLEG): container finished" podID="de020384-2f15-42b1-9e1f-3792dbe10578" containerID="1b86e5870ff1095bfa9fd1a472ab3be2f48f9775a16a2a60eed14d0cc101585b" exitCode=0
Sep 05 03:22:46.804491 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:46.804331    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" event={"ID":"de020384-2f15-42b1-9e1f-3792dbe10578","Type":"ContainerDied","Data":"1b86e5870ff1095bfa9fd1a472ab3be2f48f9775a16a2a60eed14d0cc101585b"}
Sep 05 03:22:46.995055 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:46.995021    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:22:47.056287 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.056240    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-dshm\") pod \"de020384-2f15-42b1-9e1f-3792dbe10578\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") "
Sep 05 03:22:47.056287 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.056283    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-tmp-dir\") pod \"de020384-2f15-42b1-9e1f-3792dbe10578\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") "
Sep 05 03:22:47.056598 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.056306    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhsn2\" (UniqueName: \"kubernetes.io/projected/de020384-2f15-42b1-9e1f-3792dbe10578-kube-api-access-vhsn2\") pod \"de020384-2f15-42b1-9e1f-3792dbe10578\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") "
Sep 05 03:22:47.056598 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.056363    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de020384-2f15-42b1-9e1f-3792dbe10578-cabundle-cert\") pod \"de020384-2f15-42b1-9e1f-3792dbe10578\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") "
Sep 05 03:22:47.056598 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.056380    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-home\") pod \"de020384-2f15-42b1-9e1f-3792dbe10578\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") "
Sep 05 03:22:47.056598 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.056405    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-model-cache\") pod \"de020384-2f15-42b1-9e1f-3792dbe10578\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") "
Sep 05 03:22:47.056598 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.056433    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-kserve-provision-location\") pod \"de020384-2f15-42b1-9e1f-3792dbe10578\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") "
Sep 05 03:22:47.056598 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.056474    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de020384-2f15-42b1-9e1f-3792dbe10578-tls-certs\") pod \"de020384-2f15-42b1-9e1f-3792dbe10578\" (UID: \"de020384-2f15-42b1-9e1f-3792dbe10578\") "
Sep 05 03:22:47.058120 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.057790    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-home" (OuterVolumeSpecName: "home") pod "de020384-2f15-42b1-9e1f-3792dbe10578" (UID: "de020384-2f15-42b1-9e1f-3792dbe10578"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:22:47.058364 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.058307    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-model-cache" (OuterVolumeSpecName: "model-cache") pod "de020384-2f15-42b1-9e1f-3792dbe10578" (UID: "de020384-2f15-42b1-9e1f-3792dbe10578"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:22:47.058652 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.058619    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de020384-2f15-42b1-9e1f-3792dbe10578-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de020384-2f15-42b1-9e1f-3792dbe10578" (UID: "de020384-2f15-42b1-9e1f-3792dbe10578"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:22:47.059335 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.059294    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-dshm" (OuterVolumeSpecName: "dshm") pod "de020384-2f15-42b1-9e1f-3792dbe10578" (UID: "de020384-2f15-42b1-9e1f-3792dbe10578"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:22:47.060105 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.060070    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de020384-2f15-42b1-9e1f-3792dbe10578-kube-api-access-vhsn2" (OuterVolumeSpecName: "kube-api-access-vhsn2") pod "de020384-2f15-42b1-9e1f-3792dbe10578" (UID: "de020384-2f15-42b1-9e1f-3792dbe10578"). InnerVolumeSpecName "kube-api-access-vhsn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:22:47.061224 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.061196    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de020384-2f15-42b1-9e1f-3792dbe10578-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de020384-2f15-42b1-9e1f-3792dbe10578" (UID: "de020384-2f15-42b1-9e1f-3792dbe10578"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:22:47.068725 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.068690    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de020384-2f15-42b1-9e1f-3792dbe10578" (UID: "de020384-2f15-42b1-9e1f-3792dbe10578"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:22:47.072774 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.072737    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de020384-2f15-42b1-9e1f-3792dbe10578" (UID: "de020384-2f15-42b1-9e1f-3792dbe10578"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:22:47.158383 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.158343    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:22:47.158383 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.158388    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:22:47.158383 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.158406    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhsn2\" (UniqueName: \"kubernetes.io/projected/de020384-2f15-42b1-9e1f-3792dbe10578-kube-api-access-vhsn2\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:22:47.158822 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.158421    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de020384-2f15-42b1-9e1f-3792dbe10578-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:22:47.158822 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.158463    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:22:47.158822 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.158478    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:22:47.158822 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.158492    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de020384-2f15-42b1-9e1f-3792dbe10578-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:22:47.158822 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.158506    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de020384-2f15-42b1-9e1f-3792dbe10578-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:22:47.812628 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.812588    2557 generic.go:356] "Generic (PLEG): container finished" podID="de020384-2f15-42b1-9e1f-3792dbe10578" containerID="70d1ba9d342cd641a8b4e7c15b54372ae4b2c0720cb09271e5216947e4bf012a" exitCode=0
Sep 05 03:22:47.812850 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.812653    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"
Sep 05 03:22:47.812850 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.812675    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" event={"ID":"de020384-2f15-42b1-9e1f-3792dbe10578","Type":"ContainerDied","Data":"70d1ba9d342cd641a8b4e7c15b54372ae4b2c0720cb09271e5216947e4bf012a"}
Sep 05 03:22:47.812850 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.812740    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk" event={"ID":"de020384-2f15-42b1-9e1f-3792dbe10578","Type":"ContainerDied","Data":"9c8ede2d823d1e6252a3566c6c52af158c63316f3f511af4b63f963aa07f6372"}
Sep 05 03:22:47.812850 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.812774    2557 scope.go:117] "RemoveContainer" containerID="1b86e5870ff1095bfa9fd1a472ab3be2f48f9775a16a2a60eed14d0cc101585b"
Sep 05 03:22:47.824499 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.824474    2557 scope.go:117] "RemoveContainer" containerID="1529ef563eca81492e70f50d6e9d8265fea5207e15f73d32e764d2a71e1ce3b4"
Sep 05 03:22:47.834652 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.834630    2557 scope.go:117] "RemoveContainer" containerID="70d1ba9d342cd641a8b4e7c15b54372ae4b2c0720cb09271e5216947e4bf012a"
Sep 05 03:22:47.838698 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.838671    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"]
Sep 05 03:22:47.844918 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.844886    2557 scope.go:117] "RemoveContainer" containerID="1b86e5870ff1095bfa9fd1a472ab3be2f48f9775a16a2a60eed14d0cc101585b"
Sep 05 03:22:47.845272 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:22:47.845248    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b86e5870ff1095bfa9fd1a472ab3be2f48f9775a16a2a60eed14d0cc101585b\": container with ID starting with 1b86e5870ff1095bfa9fd1a472ab3be2f48f9775a16a2a60eed14d0cc101585b not found: ID does not exist" containerID="1b86e5870ff1095bfa9fd1a472ab3be2f48f9775a16a2a60eed14d0cc101585b"
Sep 05 03:22:47.845364 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.845281    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b86e5870ff1095bfa9fd1a472ab3be2f48f9775a16a2a60eed14d0cc101585b"} err="failed to get container status \"1b86e5870ff1095bfa9fd1a472ab3be2f48f9775a16a2a60eed14d0cc101585b\": rpc error: code = NotFound desc = could not find container \"1b86e5870ff1095bfa9fd1a472ab3be2f48f9775a16a2a60eed14d0cc101585b\": container with ID starting with 1b86e5870ff1095bfa9fd1a472ab3be2f48f9775a16a2a60eed14d0cc101585b not found: ID does not exist"
Sep 05 03:22:47.845364 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.845306    2557 scope.go:117] "RemoveContainer" containerID="1529ef563eca81492e70f50d6e9d8265fea5207e15f73d32e764d2a71e1ce3b4"
Sep 05 03:22:47.845615 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.845594    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7fb469c8d8xkrbk"]
Sep 05 03:22:47.845681 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:22:47.845602    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1529ef563eca81492e70f50d6e9d8265fea5207e15f73d32e764d2a71e1ce3b4\": container with ID starting with 1529ef563eca81492e70f50d6e9d8265fea5207e15f73d32e764d2a71e1ce3b4 not found: ID does not exist" containerID="1529ef563eca81492e70f50d6e9d8265fea5207e15f73d32e764d2a71e1ce3b4"
Sep 05 03:22:47.845681 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.845631    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1529ef563eca81492e70f50d6e9d8265fea5207e15f73d32e764d2a71e1ce3b4"} err="failed to get container status \"1529ef563eca81492e70f50d6e9d8265fea5207e15f73d32e764d2a71e1ce3b4\": rpc error: code = NotFound desc = could not find container \"1529ef563eca81492e70f50d6e9d8265fea5207e15f73d32e764d2a71e1ce3b4\": container with ID starting with 1529ef563eca81492e70f50d6e9d8265fea5207e15f73d32e764d2a71e1ce3b4 not found: ID does not exist"
Sep 05 03:22:47.845681 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.845647    2557 scope.go:117] "RemoveContainer" containerID="70d1ba9d342cd641a8b4e7c15b54372ae4b2c0720cb09271e5216947e4bf012a"
Sep 05 03:22:47.845896 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:22:47.845880    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d1ba9d342cd641a8b4e7c15b54372ae4b2c0720cb09271e5216947e4bf012a\": container with ID starting with 70d1ba9d342cd641a8b4e7c15b54372ae4b2c0720cb09271e5216947e4bf012a not found: ID does not exist" containerID="70d1ba9d342cd641a8b4e7c15b54372ae4b2c0720cb09271e5216947e4bf012a"
Sep 05 03:22:47.845949 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:47.845900    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d1ba9d342cd641a8b4e7c15b54372ae4b2c0720cb09271e5216947e4bf012a"} err="failed to get container status \"70d1ba9d342cd641a8b4e7c15b54372ae4b2c0720cb09271e5216947e4bf012a\": rpc error: code = NotFound desc = could not find container \"70d1ba9d342cd641a8b4e7c15b54372ae4b2c0720cb09271e5216947e4bf012a\": container with ID starting with 70d1ba9d342cd641a8b4e7c15b54372ae4b2c0720cb09271e5216947e4bf012a not found: ID does not exist"
Sep 05 03:22:48.991472 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:48.991412    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" path="/var/lib/kubelet/pods/de020384-2f15-42b1-9e1f-3792dbe10578/volumes"
Sep 05 03:22:52.292910 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:52.292875    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-7898b797f6-5twpx_819bf449-908c-47ac-85d5-917db6e76ee1/storage-initializer/0.log"
Sep 05 03:22:52.304511 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:52.304476    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-7898b797f6-5twpx_819bf449-908c-47ac-85d5-917db6e76ee1/main/0.log"
Sep 05 03:22:53.813737 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:53.813689    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"]
Sep 05 03:22:53.838279 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:22:53.838240    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:23:08.840876 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:08.840805    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" containerName="main" containerID="cri-o://a06ca1968677d260a6432d3478dd720d4d3120455cb44da6b484c871c862762d" gracePeriod=45
Sep 05 03:23:09.493537 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.493508    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:23:09.675555 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.675512    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819bf449-908c-47ac-85d5-917db6e76ee1-tls-certs\") pod \"819bf449-908c-47ac-85d5-917db6e76ee1\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") "
Sep 05 03:23:09.675555 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.675570    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-kserve-provision-location\") pod \"819bf449-908c-47ac-85d5-917db6e76ee1\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") "
Sep 05 03:23:09.675837 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.675613    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxrxs\" (UniqueName: \"kubernetes.io/projected/819bf449-908c-47ac-85d5-917db6e76ee1-kube-api-access-cxrxs\") pod \"819bf449-908c-47ac-85d5-917db6e76ee1\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") "
Sep 05 03:23:09.675837 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.675647    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-home\") pod \"819bf449-908c-47ac-85d5-917db6e76ee1\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") "
Sep 05 03:23:09.675837 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.675663    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-tmp-dir\") pod \"819bf449-908c-47ac-85d5-917db6e76ee1\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") "
Sep 05 03:23:09.675837 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.675707    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-model-cache\") pod \"819bf449-908c-47ac-85d5-917db6e76ee1\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") "
Sep 05 03:23:09.675837 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.675750    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/819bf449-908c-47ac-85d5-917db6e76ee1-cabundle-cert\") pod \"819bf449-908c-47ac-85d5-917db6e76ee1\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") "
Sep 05 03:23:09.675837 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.675815    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-dshm\") pod \"819bf449-908c-47ac-85d5-917db6e76ee1\" (UID: \"819bf449-908c-47ac-85d5-917db6e76ee1\") "
Sep 05 03:23:09.676161 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.676104    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-model-cache" (OuterVolumeSpecName: "model-cache") pod "819bf449-908c-47ac-85d5-917db6e76ee1" (UID: "819bf449-908c-47ac-85d5-917db6e76ee1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:23:09.676341 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.676314    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819bf449-908c-47ac-85d5-917db6e76ee1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "819bf449-908c-47ac-85d5-917db6e76ee1" (UID: "819bf449-908c-47ac-85d5-917db6e76ee1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:23:09.676341 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.676330    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-home" (OuterVolumeSpecName: "home") pod "819bf449-908c-47ac-85d5-917db6e76ee1" (UID: "819bf449-908c-47ac-85d5-917db6e76ee1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:23:09.678429 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.678398    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/819bf449-908c-47ac-85d5-917db6e76ee1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "819bf449-908c-47ac-85d5-917db6e76ee1" (UID: "819bf449-908c-47ac-85d5-917db6e76ee1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:23:09.678429 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.678413    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819bf449-908c-47ac-85d5-917db6e76ee1-kube-api-access-cxrxs" (OuterVolumeSpecName: "kube-api-access-cxrxs") pod "819bf449-908c-47ac-85d5-917db6e76ee1" (UID: "819bf449-908c-47ac-85d5-917db6e76ee1"). InnerVolumeSpecName "kube-api-access-cxrxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:23:09.678579 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.678508    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-dshm" (OuterVolumeSpecName: "dshm") pod "819bf449-908c-47ac-85d5-917db6e76ee1" (UID: "819bf449-908c-47ac-85d5-917db6e76ee1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:23:09.687897 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.687867    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "819bf449-908c-47ac-85d5-917db6e76ee1" (UID: "819bf449-908c-47ac-85d5-917db6e76ee1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:23:09.698551 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.698505    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "819bf449-908c-47ac-85d5-917db6e76ee1" (UID: "819bf449-908c-47ac-85d5-917db6e76ee1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:23:09.777166 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.777120    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:23:09.777166 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.777161    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:23:09.777166 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.777173    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:23:09.777404 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.777181    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/819bf449-908c-47ac-85d5-917db6e76ee1-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:23:09.777404 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.777190    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:23:09.777404 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.777198    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819bf449-908c-47ac-85d5-917db6e76ee1-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:23:09.777404 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.777207    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/819bf449-908c-47ac-85d5-917db6e76ee1-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:23:09.777404 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.777215    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxrxs\" (UniqueName: \"kubernetes.io/projected/819bf449-908c-47ac-85d5-917db6e76ee1-kube-api-access-cxrxs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:23:09.944512 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.944398    2557 generic.go:356] "Generic (PLEG): container finished" podID="819bf449-908c-47ac-85d5-917db6e76ee1" containerID="a06ca1968677d260a6432d3478dd720d4d3120455cb44da6b484c871c862762d" exitCode=0
Sep 05 03:23:09.944512 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.944477    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"
Sep 05 03:23:09.944512 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.944487    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" event={"ID":"819bf449-908c-47ac-85d5-917db6e76ee1","Type":"ContainerDied","Data":"a06ca1968677d260a6432d3478dd720d4d3120455cb44da6b484c871c862762d"}
Sep 05 03:23:09.945071 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.944544    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx" event={"ID":"819bf449-908c-47ac-85d5-917db6e76ee1","Type":"ContainerDied","Data":"e17521f19fa561d840406714177e0e746cdc7104713f53baeb343674968c03f5"}
Sep 05 03:23:09.945071 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.944567    2557 scope.go:117] "RemoveContainer" containerID="a06ca1968677d260a6432d3478dd720d4d3120455cb44da6b484c871c862762d"
Sep 05 03:23:09.953832 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.953704    2557 scope.go:117] "RemoveContainer" containerID="f5cb19fb04ebc67ecc04dde2261559a9996624afb17a706739c435c54423a6bc"
Sep 05 03:23:09.962484 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.962466    2557 scope.go:117] "RemoveContainer" containerID="a06ca1968677d260a6432d3478dd720d4d3120455cb44da6b484c871c862762d"
Sep 05 03:23:09.962771 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:23:09.962752    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06ca1968677d260a6432d3478dd720d4d3120455cb44da6b484c871c862762d\": container with ID starting with a06ca1968677d260a6432d3478dd720d4d3120455cb44da6b484c871c862762d not found: ID does not exist" containerID="a06ca1968677d260a6432d3478dd720d4d3120455cb44da6b484c871c862762d"
Sep 05 03:23:09.962816 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.962779    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06ca1968677d260a6432d3478dd720d4d3120455cb44da6b484c871c862762d"} err="failed to get container status \"a06ca1968677d260a6432d3478dd720d4d3120455cb44da6b484c871c862762d\": rpc error: code = NotFound desc = could not find container \"a06ca1968677d260a6432d3478dd720d4d3120455cb44da6b484c871c862762d\": container with ID starting with a06ca1968677d260a6432d3478dd720d4d3120455cb44da6b484c871c862762d not found: ID does not exist"
Sep 05 03:23:09.962816 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.962796    2557 scope.go:117] "RemoveContainer" containerID="f5cb19fb04ebc67ecc04dde2261559a9996624afb17a706739c435c54423a6bc"
Sep 05 03:23:09.963063 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:23:09.963043    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5cb19fb04ebc67ecc04dde2261559a9996624afb17a706739c435c54423a6bc\": container with ID starting with f5cb19fb04ebc67ecc04dde2261559a9996624afb17a706739c435c54423a6bc not found: ID does not exist" containerID="f5cb19fb04ebc67ecc04dde2261559a9996624afb17a706739c435c54423a6bc"
Sep 05 03:23:09.963135 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.963073    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5cb19fb04ebc67ecc04dde2261559a9996624afb17a706739c435c54423a6bc"} err="failed to get container status \"f5cb19fb04ebc67ecc04dde2261559a9996624afb17a706739c435c54423a6bc\": rpc error: code = NotFound desc = could not find container \"f5cb19fb04ebc67ecc04dde2261559a9996624afb17a706739c435c54423a6bc\": container with ID starting with f5cb19fb04ebc67ecc04dde2261559a9996624afb17a706739c435c54423a6bc not found: ID does not exist"
Sep 05 03:23:09.968504 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.968480    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"]
Sep 05 03:23:09.972874 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:09.972850    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7898b797f6-5twpx"]
Sep 05 03:23:10.990724 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:10.990686    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" path="/var/lib/kubelet/pods/819bf449-908c-47ac-85d5-917db6e76ee1/volumes"
Sep 05 03:23:14.055292 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:14.055249    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:23:14.154588 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:14.154545    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:23:14.254822 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:14.254790    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:23:17.331521 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:17.331489    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:23:17.345116 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:17.345090    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:23:18.797666 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.797625    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"]
Sep 05 03:23:18.798269 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.797972    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main"
Sep 05 03:23:18.798269 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.797984    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main"
Sep 05 03:23:18.798269 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.797994    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="llm-d-routing-sidecar"
Sep 05 03:23:18.798269 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.798000    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="llm-d-routing-sidecar"
Sep 05 03:23:18.798269 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.798007    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="storage-initializer"
Sep 05 03:23:18.798269 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.798013    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="storage-initializer"
Sep 05 03:23:18.798269 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.798020    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" containerName="main"
Sep 05 03:23:18.798269 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.798026    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" containerName="main"
Sep 05 03:23:18.798269 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.798033    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" containerName="storage-initializer"
Sep 05 03:23:18.798269 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.798038    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" containerName="storage-initializer"
Sep 05 03:23:18.798269 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.798093    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="llm-d-routing-sidecar"
Sep 05 03:23:18.798269 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.798099    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="819bf449-908c-47ac-85d5-917db6e76ee1" containerName="main"
Sep 05 03:23:18.798269 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.798106    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="de020384-2f15-42b1-9e1f-3792dbe10578" containerName="main"
Sep 05 03:23:18.803283 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.803253    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.805814 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.805737    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 05 03:23:18.805976 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.805737    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 05 03:23:18.806044 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.805860    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:23:18.807134 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.806150    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-xhc4l\""
Sep 05 03:23:18.807134 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.805867    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 05 03:23:18.807134 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.805919    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 05 03:23:18.820155 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.820128    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"]
Sep 05 03:23:18.843702 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.843670    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:23:18.857826 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.857801    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.857950 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.857838    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d2a6ec-2885-4592-833d-a48e57b275c8-tls-certs\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.857950 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.857868    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-dshm\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.858050 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.857939    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkbrb\" (UniqueName: \"kubernetes.io/projected/40d2a6ec-2885-4592-833d-a48e57b275c8-kube-api-access-lkbrb\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.858050 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.857992    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40d2a6ec-2885-4592-833d-a48e57b275c8-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.858050 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.858019    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-model-cache\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.858173 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.858057    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.858173 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.858140    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-home\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.958900 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.958863    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-home\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.959087 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.958912    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.959087 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.958930    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d2a6ec-2885-4592-833d-a48e57b275c8-tls-certs\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.959087 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.958967    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-dshm\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.959087 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.958990    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkbrb\" (UniqueName: \"kubernetes.io/projected/40d2a6ec-2885-4592-833d-a48e57b275c8-kube-api-access-lkbrb\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.959087 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.959014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40d2a6ec-2885-4592-833d-a48e57b275c8-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.959087 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.959053    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-model-cache\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.959456 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.959109    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.959545 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.959521    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.959605 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.959575    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.959991 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.959941    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-home\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.960086 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.960009    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-model-cache\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.960453 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.960413    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40d2a6ec-2885-4592-833d-a48e57b275c8-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.962162 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.962131    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-dshm\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.962567 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.962548    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d2a6ec-2885-4592-833d-a48e57b275c8-tls-certs\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:18.966978 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:18.966956    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkbrb\" (UniqueName: \"kubernetes.io/projected/40d2a6ec-2885-4592-833d-a48e57b275c8-kube-api-access-lkbrb\") pod \"lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:19.120284 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:19.120249    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:19.415163 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:19.415128    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"]
Sep 05 03:23:19.416536 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:23:19.416506    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d2a6ec_2885_4592_833d_a48e57b275c8.slice/crio-dfdfbd77ae4e1952f5970fd25e043afb087d6546873c6c6e05f2c7ce867ea605 WatchSource:0}: Error finding container dfdfbd77ae4e1952f5970fd25e043afb087d6546873c6c6e05f2c7ce867ea605: Status 404 returned error can't find the container with id dfdfbd77ae4e1952f5970fd25e043afb087d6546873c6c6e05f2c7ce867ea605
Sep 05 03:23:19.999733 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:19.999689    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" event={"ID":"40d2a6ec-2885-4592-833d-a48e57b275c8","Type":"ContainerStarted","Data":"91860937151f41a7a086a9a697c98f69c7d94dc6f7a9bd64cc0de9db2a3f94d5"}
Sep 05 03:23:20.000226 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:19.999762    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" event={"ID":"40d2a6ec-2885-4592-833d-a48e57b275c8","Type":"ContainerStarted","Data":"dfdfbd77ae4e1952f5970fd25e043afb087d6546873c6c6e05f2c7ce867ea605"}
Sep 05 03:23:20.990787 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:20.990748    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:23:21.093283 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:21.093247    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:23:22.493351 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:22.493314    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:23:22.590319 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:22.590278    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:23:23.017103 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:23.017066    2557 generic.go:356] "Generic (PLEG): container finished" podID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerID="91860937151f41a7a086a9a697c98f69c7d94dc6f7a9bd64cc0de9db2a3f94d5" exitCode=0
Sep 05 03:23:23.017239 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:23.017132    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" event={"ID":"40d2a6ec-2885-4592-833d-a48e57b275c8","Type":"ContainerDied","Data":"91860937151f41a7a086a9a697c98f69c7d94dc6f7a9bd64cc0de9db2a3f94d5"}
Sep 05 03:23:24.024153 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:24.024113    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" event={"ID":"40d2a6ec-2885-4592-833d-a48e57b275c8","Type":"ContainerStarted","Data":"e56366f2efe6a02950d67a5ba4ec7015bbb3b184b97fd6f96d60ace1cb07b995"}
Sep 05 03:23:24.047793 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:24.047730    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" podStartSLOduration=2.567288969 podStartE2EDuration="6.047708769s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.4804198s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:23:24.045683017 +0000 UTC m=+3907.600839323" watchObservedRunningTime="2026-09-05 03:23:24.047708769 +0000 UTC m=+3907.602865080"
Sep 05 03:23:24.121326 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:24.121293    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:26.988196 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:26.988156    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:23:29.121337 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:29.121298    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:23:29.123096 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:29.123060    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Sep 05 03:23:32.837639 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:32.837599    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:23:32.938747 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:32.938701    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:23:36.140572 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.140535    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"]
Sep 05 03:23:36.144355 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.144331    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.147330 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.147302    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:23:36.147509 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.147374    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 05 03:23:36.147509 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.147399    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-b2587\""
Sep 05 03:23:36.147776 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.147758    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 05 03:23:36.147776 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.147768    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 05 03:23:36.147938 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.147760    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-cbq4n\""
Sep 05 03:23:36.148225 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.148207    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 05 03:23:36.165060 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.165027    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"]
Sep 05 03:23:36.203061 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.203026    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:23:36.220511 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.220473    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.220691 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.220527    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-dshm\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.220691 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.220587    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-model-cache\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.220691 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.220624    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-home\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.220691 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.220651    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjh6k\" (UniqueName: \"kubernetes.io/projected/c155ba4f-db57-4c9a-a66d-b82c74347162-kube-api-access-wjh6k\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.220915 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.220734    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c155ba4f-db57-4c9a-a66d-b82c74347162-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.220915 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.220771    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.220915 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.220805    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c155ba4f-db57-4c9a-a66d-b82c74347162-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.322232 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.322193    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.322400 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.322243    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-dshm\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.322400 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.322276    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-model-cache\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.322400 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.322306    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-home\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.322400 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.322321    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjh6k\" (UniqueName: \"kubernetes.io/projected/c155ba4f-db57-4c9a-a66d-b82c74347162-kube-api-access-wjh6k\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.322686 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.322407    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c155ba4f-db57-4c9a-a66d-b82c74347162-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.322686 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.322480    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.322686 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.322518    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c155ba4f-db57-4c9a-a66d-b82c74347162-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.322851 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.322710    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.322966 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.322942    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-model-cache\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.323035 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.322985    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.323261 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.323235    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-home\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.323408 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.323284    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c155ba4f-db57-4c9a-a66d-b82c74347162-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.325118 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.325088    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c155ba4f-db57-4c9a-a66d-b82c74347162-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.325251 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.325116    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-dshm\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.330145 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.330119    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjh6k\" (UniqueName: \"kubernetes.io/projected/c155ba4f-db57-4c9a-a66d-b82c74347162-kube-api-access-wjh6k\") pod \"custom-route-timeout-pd-test-kserve-76448789f-7qbks\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.455156 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.455047    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:36.456847 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.456810    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:23:36.751334 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:36.751245    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"]
Sep 05 03:23:36.755403 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:23:36.755371    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc155ba4f_db57_4c9a_a66d_b82c74347162.slice/crio-8d6085c3d218c99d81587a54da7366becd5ae430fe0e05240d2ce406817c18f0 WatchSource:0}: Error finding container 8d6085c3d218c99d81587a54da7366becd5ae430fe0e05240d2ce406817c18f0: Status 404 returned error can't find the container with id 8d6085c3d218c99d81587a54da7366becd5ae430fe0e05240d2ce406817c18f0
Sep 05 03:23:37.105126 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:37.105083    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" event={"ID":"c155ba4f-db57-4c9a-a66d-b82c74347162","Type":"ContainerStarted","Data":"0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909"}
Sep 05 03:23:37.105334 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:37.105155    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" event={"ID":"c155ba4f-db57-4c9a-a66d-b82c74347162","Type":"ContainerStarted","Data":"8d6085c3d218c99d81587a54da7366becd5ae430fe0e05240d2ce406817c18f0"}
Sep 05 03:23:37.105334 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:37.105197    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:38.112451 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:38.112403    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" event={"ID":"c155ba4f-db57-4c9a-a66d-b82c74347162","Type":"ContainerStarted","Data":"560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb"}
Sep 05 03:23:39.120777 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:39.120740    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Sep 05 03:23:40.125037 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:40.124996    2557 generic.go:356] "Generic (PLEG): container finished" podID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerID="560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb" exitCode=0
Sep 05 03:23:40.125431 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:40.125072    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" event={"ID":"c155ba4f-db57-4c9a-a66d-b82c74347162","Type":"ContainerDied","Data":"560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb"}
Sep 05 03:23:41.133994 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:41.133954    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" event={"ID":"c155ba4f-db57-4c9a-a66d-b82c74347162","Type":"ContainerStarted","Data":"efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2"}
Sep 05 03:23:41.160676 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:41.160609    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podStartSLOduration=2.646451615 podStartE2EDuration="5.160587963s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.514136348s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:23:41.158742213 +0000 UTC m=+3924.713898521" watchObservedRunningTime="2026-09-05 03:23:41.160587963 +0000 UTC m=+3924.715744274"
Sep 05 03:23:46.455660 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:46.455610    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:46.456191 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:46.455717    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:46.457365 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:46.457320    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused"
Sep 05 03:23:46.486755 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:46.486720    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:23:49.121178 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:49.121124    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Sep 05 03:23:56.456270 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:56.456218    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused"
Sep 05 03:23:59.121618 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:23:59.121558    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Sep 05 03:24:06.456452 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:24:06.455748    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused"
Sep 05 03:24:09.120880 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:24:09.120771    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Sep 05 03:24:14.753422 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:24:14.753374    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:24:16.455887 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:24:16.455838    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused"
Sep 05 03:24:19.121769 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:24:19.121724    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Sep 05 03:24:26.455912 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:24:26.455856    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused"
Sep 05 03:24:29.120713 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:24:29.120669    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Sep 05 03:24:36.456325 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:24:36.456267    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused"
Sep 05 03:24:39.121752 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:24:39.121696    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Sep 05 03:24:46.455885 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:24:46.455816    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused"
Sep 05 03:24:49.130700 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:24:49.130654    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:24:49.138621 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:24:49.138586    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:24:56.455707 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:24:56.455655    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused"
Sep 05 03:25:06.456571 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:06.456507    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused"
Sep 05 03:25:16.455642 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:16.455585    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused"
Sep 05 03:25:26.456118 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:26.456045    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused"
Sep 05 03:25:35.324844 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:35.324793    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf_40d2a6ec-2885-4592-833d-a48e57b275c8/storage-initializer/0.log"
Sep 05 03:25:35.336368 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:35.336336    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf_40d2a6ec-2885-4592-833d-a48e57b275c8/main/0.log"
Sep 05 03:25:36.455893 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:36.455835    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused"
Sep 05 03:25:36.483013 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:36.482973    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"]
Sep 05 03:25:36.502658 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:36.502610    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:25:46.456111 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:46.455991    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused"
Sep 05 03:25:51.507838 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:51.507787    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="main" containerID="cri-o://e56366f2efe6a02950d67a5ba4ec7015bbb3b184b97fd6f96d60ace1cb07b995" gracePeriod=45
Sep 05 03:25:52.131994 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.131944    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Sep 05 03:25:52.561591 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.561550    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:25:52.642030 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.641989    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-tmp-dir\") pod \"40d2a6ec-2885-4592-833d-a48e57b275c8\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") "
Sep 05 03:25:52.642030 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.642033    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-kserve-provision-location\") pod \"40d2a6ec-2885-4592-833d-a48e57b275c8\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") "
Sep 05 03:25:52.642317 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.642073    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-model-cache\") pod \"40d2a6ec-2885-4592-833d-a48e57b275c8\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") "
Sep 05 03:25:52.642317 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.642105    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-dshm\") pod \"40d2a6ec-2885-4592-833d-a48e57b275c8\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") "
Sep 05 03:25:52.642317 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.642124    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-home\") pod \"40d2a6ec-2885-4592-833d-a48e57b275c8\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") "
Sep 05 03:25:52.642317 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.642156    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkbrb\" (UniqueName: \"kubernetes.io/projected/40d2a6ec-2885-4592-833d-a48e57b275c8-kube-api-access-lkbrb\") pod \"40d2a6ec-2885-4592-833d-a48e57b275c8\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") "
Sep 05 03:25:52.642317 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.642205    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d2a6ec-2885-4592-833d-a48e57b275c8-tls-certs\") pod \"40d2a6ec-2885-4592-833d-a48e57b275c8\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") "
Sep 05 03:25:52.642317 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.642237    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40d2a6ec-2885-4592-833d-a48e57b275c8-cabundle-cert\") pod \"40d2a6ec-2885-4592-833d-a48e57b275c8\" (UID: \"40d2a6ec-2885-4592-833d-a48e57b275c8\") "
Sep 05 03:25:52.642699 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.642511    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "40d2a6ec-2885-4592-833d-a48e57b275c8" (UID: "40d2a6ec-2885-4592-833d-a48e57b275c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:25:52.642699 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.642522    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-home" (OuterVolumeSpecName: "home") pod "40d2a6ec-2885-4592-833d-a48e57b275c8" (UID: "40d2a6ec-2885-4592-833d-a48e57b275c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:25:52.643064 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.642885    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d2a6ec-2885-4592-833d-a48e57b275c8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "40d2a6ec-2885-4592-833d-a48e57b275c8" (UID: "40d2a6ec-2885-4592-833d-a48e57b275c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:25:52.643064 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.643000    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:25:52.643064 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.643020    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:25:52.643064 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.643036    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40d2a6ec-2885-4592-833d-a48e57b275c8-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:25:52.644900 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.644874    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d2a6ec-2885-4592-833d-a48e57b275c8-kube-api-access-lkbrb" (OuterVolumeSpecName: "kube-api-access-lkbrb") pod "40d2a6ec-2885-4592-833d-a48e57b275c8" (UID: "40d2a6ec-2885-4592-833d-a48e57b275c8"). InnerVolumeSpecName "kube-api-access-lkbrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:25:52.645081 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.645053    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-dshm" (OuterVolumeSpecName: "dshm") pod "40d2a6ec-2885-4592-833d-a48e57b275c8" (UID: "40d2a6ec-2885-4592-833d-a48e57b275c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:25:52.645212 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.645181    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d2a6ec-2885-4592-833d-a48e57b275c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40d2a6ec-2885-4592-833d-a48e57b275c8" (UID: "40d2a6ec-2885-4592-833d-a48e57b275c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:25:52.654868 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.654837    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40d2a6ec-2885-4592-833d-a48e57b275c8" (UID: "40d2a6ec-2885-4592-833d-a48e57b275c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:25:52.673721 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.673641    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40d2a6ec-2885-4592-833d-a48e57b275c8" (UID: "40d2a6ec-2885-4592-833d-a48e57b275c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:25:52.743828 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.743788    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:25:52.743828 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.743825    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkbrb\" (UniqueName: \"kubernetes.io/projected/40d2a6ec-2885-4592-833d-a48e57b275c8-kube-api-access-lkbrb\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:25:52.743828 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.743837    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d2a6ec-2885-4592-833d-a48e57b275c8-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:25:52.744062 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.743846    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:25:52.744062 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.743854    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d2a6ec-2885-4592-833d-a48e57b275c8-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:25:52.976862 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.976767    2557 generic.go:356] "Generic (PLEG): container finished" podID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerID="e56366f2efe6a02950d67a5ba4ec7015bbb3b184b97fd6f96d60ace1cb07b995" exitCode=0
Sep 05 03:25:52.976862 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.976818    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"
Sep 05 03:25:52.976862 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.976852    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" event={"ID":"40d2a6ec-2885-4592-833d-a48e57b275c8","Type":"ContainerDied","Data":"e56366f2efe6a02950d67a5ba4ec7015bbb3b184b97fd6f96d60ace1cb07b995"}
Sep 05 03:25:52.977102 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.976910    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf" event={"ID":"40d2a6ec-2885-4592-833d-a48e57b275c8","Type":"ContainerDied","Data":"dfdfbd77ae4e1952f5970fd25e043afb087d6546873c6c6e05f2c7ce867ea605"}
Sep 05 03:25:52.977102 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.976933    2557 scope.go:117] "RemoveContainer" containerID="e56366f2efe6a02950d67a5ba4ec7015bbb3b184b97fd6f96d60ace1cb07b995"
Sep 05 03:25:52.988077 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.988053    2557 scope.go:117] "RemoveContainer" containerID="91860937151f41a7a086a9a697c98f69c7d94dc6f7a9bd64cc0de9db2a3f94d5"
Sep 05 03:25:52.999699 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:52.999674    2557 scope.go:117] "RemoveContainer" containerID="e56366f2efe6a02950d67a5ba4ec7015bbb3b184b97fd6f96d60ace1cb07b995"
Sep 05 03:25:53.000083 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:25:53.000052    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56366f2efe6a02950d67a5ba4ec7015bbb3b184b97fd6f96d60ace1cb07b995\": container with ID starting with e56366f2efe6a02950d67a5ba4ec7015bbb3b184b97fd6f96d60ace1cb07b995 not found: ID does not exist" containerID="e56366f2efe6a02950d67a5ba4ec7015bbb3b184b97fd6f96d60ace1cb07b995"
Sep 05 03:25:53.000193 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:53.000095    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56366f2efe6a02950d67a5ba4ec7015bbb3b184b97fd6f96d60ace1cb07b995"} err="failed to get container status \"e56366f2efe6a02950d67a5ba4ec7015bbb3b184b97fd6f96d60ace1cb07b995\": rpc error: code = NotFound desc = could not find container \"e56366f2efe6a02950d67a5ba4ec7015bbb3b184b97fd6f96d60ace1cb07b995\": container with ID starting with e56366f2efe6a02950d67a5ba4ec7015bbb3b184b97fd6f96d60ace1cb07b995 not found: ID does not exist"
Sep 05 03:25:53.000193 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:53.000123    2557 scope.go:117] "RemoveContainer" containerID="91860937151f41a7a086a9a697c98f69c7d94dc6f7a9bd64cc0de9db2a3f94d5"
Sep 05 03:25:53.000527 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:25:53.000497    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91860937151f41a7a086a9a697c98f69c7d94dc6f7a9bd64cc0de9db2a3f94d5\": container with ID starting with 91860937151f41a7a086a9a697c98f69c7d94dc6f7a9bd64cc0de9db2a3f94d5 not found: ID does not exist" containerID="91860937151f41a7a086a9a697c98f69c7d94dc6f7a9bd64cc0de9db2a3f94d5"
Sep 05 03:25:53.000678 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:53.000533    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91860937151f41a7a086a9a697c98f69c7d94dc6f7a9bd64cc0de9db2a3f94d5"} err="failed to get container status \"91860937151f41a7a086a9a697c98f69c7d94dc6f7a9bd64cc0de9db2a3f94d5\": rpc error: code = NotFound desc = could not find container \"91860937151f41a7a086a9a697c98f69c7d94dc6f7a9bd64cc0de9db2a3f94d5\": container with ID starting with 91860937151f41a7a086a9a697c98f69c7d94dc6f7a9bd64cc0de9db2a3f94d5 not found: ID does not exist"
Sep 05 03:25:53.002493 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:53.002420    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"]
Sep 05 03:25:53.005482 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:53.005459    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57b6d74fd9-gn7qf"]
Sep 05 03:25:54.999879 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:54.999831    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" path="/var/lib/kubelet/pods/40d2a6ec-2885-4592-833d-a48e57b275c8/volumes"
Sep 05 03:25:56.456140 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:56.456083    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8001/health\": dial tcp 10.133.0.102:8001: connect: connection refused"
Sep 05 03:25:56.640694 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:56.640651    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:25:56.762699 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:25:56.762580    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:26:02.629864 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:02.629821    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:26:02.729574 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:02.729530    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:26:06.465652 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:06.465614    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:26:06.478146 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:06.478114    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:26:14.818171 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.818136    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"]
Sep 05 03:26:14.819653 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.819610    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="main"
Sep 05 03:26:14.819847 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.819829    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="main"
Sep 05 03:26:14.819986 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.819971    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="storage-initializer"
Sep 05 03:26:14.820079 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.820068    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="storage-initializer"
Sep 05 03:26:14.820426 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.820411    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d2a6ec-2885-4592-833d-a48e57b275c8" containerName="main"
Sep 05 03:26:14.823781 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.823759    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.826769 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.826738    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 05 03:26:14.826889 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.826861    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-6g6g9\""
Sep 05 03:26:14.827025 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.827002    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 05 03:26:14.827550 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.827530    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 05 03:26:14.827652 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.827579    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 05 03:26:14.827652 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.827590    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:26:14.830993 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.830968    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-kserve-provision-location\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.831119 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.831004    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jpnr\" (UniqueName: \"kubernetes.io/projected/b578de5c-386f-4b91-a464-5ccd113fc4eb-kube-api-access-2jpnr\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.831119 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.831031    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-tmp-dir\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.831119 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.831062    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-home\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.831119 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.831087    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b578de5c-386f-4b91-a464-5ccd113fc4eb-tls-certs\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.831119 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.831109    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b578de5c-386f-4b91-a464-5ccd113fc4eb-cabundle-cert\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.831431 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.831145    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-model-cache\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.831431 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.831176    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-dshm\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.833788 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.833759    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"]
Sep 05 03:26:14.834920 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.834805    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:26:14.932104 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.932014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-kserve-provision-location\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.932322 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.932238    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jpnr\" (UniqueName: \"kubernetes.io/projected/b578de5c-386f-4b91-a464-5ccd113fc4eb-kube-api-access-2jpnr\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.932322 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.932276    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-tmp-dir\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.932322 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.932319    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-home\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.932544 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.932485    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b578de5c-386f-4b91-a464-5ccd113fc4eb-tls-certs\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.932544 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.932526    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b578de5c-386f-4b91-a464-5ccd113fc4eb-cabundle-cert\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.932671 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.932489    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-kserve-provision-location\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.932671 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.932570    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-model-cache\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.932671 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.932610    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-dshm\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.932814 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.932686    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-tmp-dir\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.932814 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.932720    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-home\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.932993 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.932968    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-model-cache\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.933276 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.933250    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b578de5c-386f-4b91-a464-5ccd113fc4eb-cabundle-cert\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.935009 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.934989    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-dshm\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.935127 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.935053    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b578de5c-386f-4b91-a464-5ccd113fc4eb-tls-certs\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:14.940772 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:14.940744    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jpnr\" (UniqueName: \"kubernetes.io/projected/b578de5c-386f-4b91-a464-5ccd113fc4eb-kube-api-access-2jpnr\") pod \"tls-verification-test-kserve-d8f4cd5b9-db72l\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:15.065946 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:15.065900    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:26:15.136728 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:15.136628    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:15.424490 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:15.424456    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"]
Sep 05 03:26:15.426681 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:26:15.426650    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb578de5c_386f_4b91_a464_5ccd113fc4eb.slice/crio-1df58cff2cb1245690371fad6a549e5b328f0f49f84ffd6418b818ffdac3b5f9 WatchSource:0}: Error finding container 1df58cff2cb1245690371fad6a549e5b328f0f49f84ffd6418b818ffdac3b5f9: Status 404 returned error can't find the container with id 1df58cff2cb1245690371fad6a549e5b328f0f49f84ffd6418b818ffdac3b5f9
Sep 05 03:26:16.120192 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:16.120146    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" event={"ID":"b578de5c-386f-4b91-a464-5ccd113fc4eb","Type":"ContainerStarted","Data":"4bee910528a50073c9b37a4b6225ca41341058f7883b7710004a35c9489f41b8"}
Sep 05 03:26:16.120713 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:16.120201    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" event={"ID":"b578de5c-386f-4b91-a464-5ccd113fc4eb","Type":"ContainerStarted","Data":"1df58cff2cb1245690371fad6a549e5b328f0f49f84ffd6418b818ffdac3b5f9"}
Sep 05 03:26:18.132089 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:18.132053    2557 generic.go:356] "Generic (PLEG): container finished" podID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerID="4bee910528a50073c9b37a4b6225ca41341058f7883b7710004a35c9489f41b8" exitCode=0
Sep 05 03:26:18.132536 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:18.132096    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" event={"ID":"b578de5c-386f-4b91-a464-5ccd113fc4eb","Type":"ContainerDied","Data":"4bee910528a50073c9b37a4b6225ca41341058f7883b7710004a35c9489f41b8"}
Sep 05 03:26:19.139685 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:19.139644    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" event={"ID":"b578de5c-386f-4b91-a464-5ccd113fc4eb","Type":"ContainerStarted","Data":"5673f2712c00b39d13ed345a532eff15d13d1d9f8145db94ef704c265d3e182c"}
Sep 05 03:26:19.163275 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:19.163212    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" podStartSLOduration=2.712114852 podStartE2EDuration="5.163195363s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.451080511s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:26:19.160926733 +0000 UTC m=+4082.716083041" watchObservedRunningTime="2026-09-05 03:26:19.163195363 +0000 UTC m=+4082.718351671"
Sep 05 03:26:20.137630 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:20.137592    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:25.137842 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:25.137794    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:26:25.139428 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:25.139378    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused"
Sep 05 03:26:31.129050 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:31.129007    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:26:32.803642 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:32.803593    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:26:34.727139 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:34.727098    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:26:34.730283 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:34.730248    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"]
Sep 05 03:26:34.829703 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:34.829662    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:26:35.137142 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:35.137080    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused"
Sep 05 03:26:45.138017 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:45.137960    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused"
Sep 05 03:26:49.756682 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:49.756635    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main" containerID="cri-o://efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2" gracePeriod=45
Sep 05 03:26:50.368412 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.368371    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="llm-d-routing-sidecar" containerID="cri-o://0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909" gracePeriod=45
Sep 05 03:26:50.624749 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.624714    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:26:50.685134 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.685092    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjh6k\" (UniqueName: \"kubernetes.io/projected/c155ba4f-db57-4c9a-a66d-b82c74347162-kube-api-access-wjh6k\") pod \"c155ba4f-db57-4c9a-a66d-b82c74347162\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") "
Sep 05 03:26:50.685356 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.685165    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c155ba4f-db57-4c9a-a66d-b82c74347162-cabundle-cert\") pod \"c155ba4f-db57-4c9a-a66d-b82c74347162\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") "
Sep 05 03:26:50.685356 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.685197    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-kserve-provision-location\") pod \"c155ba4f-db57-4c9a-a66d-b82c74347162\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") "
Sep 05 03:26:50.685356 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.685281    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-dshm\") pod \"c155ba4f-db57-4c9a-a66d-b82c74347162\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") "
Sep 05 03:26:50.685356 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.685303    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c155ba4f-db57-4c9a-a66d-b82c74347162-tls-certs\") pod \"c155ba4f-db57-4c9a-a66d-b82c74347162\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") "
Sep 05 03:26:50.685356 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.685338    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-home\") pod \"c155ba4f-db57-4c9a-a66d-b82c74347162\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") "
Sep 05 03:26:50.685687 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.685388    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-tmp-dir\") pod \"c155ba4f-db57-4c9a-a66d-b82c74347162\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") "
Sep 05 03:26:50.685687 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.685416    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-model-cache\") pod \"c155ba4f-db57-4c9a-a66d-b82c74347162\" (UID: \"c155ba4f-db57-4c9a-a66d-b82c74347162\") "
Sep 05 03:26:50.685687 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.685623    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c155ba4f-db57-4c9a-a66d-b82c74347162-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c155ba4f-db57-4c9a-a66d-b82c74347162" (UID: "c155ba4f-db57-4c9a-a66d-b82c74347162"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:26:50.685881 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.685854    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-model-cache" (OuterVolumeSpecName: "model-cache") pod "c155ba4f-db57-4c9a-a66d-b82c74347162" (UID: "c155ba4f-db57-4c9a-a66d-b82c74347162"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:26:50.685964 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.685897    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c155ba4f-db57-4c9a-a66d-b82c74347162-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.686261 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.686216    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-home" (OuterVolumeSpecName: "home") pod "c155ba4f-db57-4c9a-a66d-b82c74347162" (UID: "c155ba4f-db57-4c9a-a66d-b82c74347162"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:26:50.687729 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.687697    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-dshm" (OuterVolumeSpecName: "dshm") pod "c155ba4f-db57-4c9a-a66d-b82c74347162" (UID: "c155ba4f-db57-4c9a-a66d-b82c74347162"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:26:50.687836 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.687772    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c155ba4f-db57-4c9a-a66d-b82c74347162-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c155ba4f-db57-4c9a-a66d-b82c74347162" (UID: "c155ba4f-db57-4c9a-a66d-b82c74347162"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:26:50.688228 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.688202    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c155ba4f-db57-4c9a-a66d-b82c74347162-kube-api-access-wjh6k" (OuterVolumeSpecName: "kube-api-access-wjh6k") pod "c155ba4f-db57-4c9a-a66d-b82c74347162" (UID: "c155ba4f-db57-4c9a-a66d-b82c74347162"). InnerVolumeSpecName "kube-api-access-wjh6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:26:50.703952 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.703902    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c155ba4f-db57-4c9a-a66d-b82c74347162" (UID: "c155ba4f-db57-4c9a-a66d-b82c74347162"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:26:50.708016 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.707984    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c155ba4f-db57-4c9a-a66d-b82c74347162" (UID: "c155ba4f-db57-4c9a-a66d-b82c74347162"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:26:50.787038 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.786992    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.787038 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.787042    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.787511 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.787059    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.787511 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.787075    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjh6k\" (UniqueName: \"kubernetes.io/projected/c155ba4f-db57-4c9a-a66d-b82c74347162-kube-api-access-wjh6k\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.787511 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.787090    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.787511 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.787105    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c155ba4f-db57-4c9a-a66d-b82c74347162-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:26:50.787511 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:50.787118    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c155ba4f-db57-4c9a-a66d-b82c74347162-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:26:51.342468 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.342413    2557 generic.go:356] "Generic (PLEG): container finished" podID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerID="efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2" exitCode=0
Sep 05 03:26:51.342468 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.342465    2557 generic.go:356] "Generic (PLEG): container finished" podID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerID="0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909" exitCode=0
Sep 05 03:26:51.342706 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.342496    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"
Sep 05 03:26:51.342706 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.342498    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" event={"ID":"c155ba4f-db57-4c9a-a66d-b82c74347162","Type":"ContainerDied","Data":"efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2"}
Sep 05 03:26:51.342706 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.342565    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" event={"ID":"c155ba4f-db57-4c9a-a66d-b82c74347162","Type":"ContainerDied","Data":"0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909"}
Sep 05 03:26:51.342706 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.342586    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks" event={"ID":"c155ba4f-db57-4c9a-a66d-b82c74347162","Type":"ContainerDied","Data":"8d6085c3d218c99d81587a54da7366becd5ae430fe0e05240d2ce406817c18f0"}
Sep 05 03:26:51.342706 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.342608    2557 scope.go:117] "RemoveContainer" containerID="efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2"
Sep 05 03:26:51.352233 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.352204    2557 scope.go:117] "RemoveContainer" containerID="560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb"
Sep 05 03:26:51.362138 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.362109    2557 scope.go:117] "RemoveContainer" containerID="0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909"
Sep 05 03:26:51.364079 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.364047    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"]
Sep 05 03:26:51.372920 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.372859    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-76448789f-7qbks"]
Sep 05 03:26:51.377542 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.377513    2557 scope.go:117] "RemoveContainer" containerID="efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2"
Sep 05 03:26:51.377899 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:26:51.377873    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2\": container with ID starting with efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2 not found: ID does not exist" containerID="efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2"
Sep 05 03:26:51.377992 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.377906    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2"} err="failed to get container status \"efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2\": rpc error: code = NotFound desc = could not find container \"efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2\": container with ID starting with efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2 not found: ID does not exist"
Sep 05 03:26:51.377992 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.377926    2557 scope.go:117] "RemoveContainer" containerID="560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb"
Sep 05 03:26:51.378215 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:26:51.378185    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb\": container with ID starting with 560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb not found: ID does not exist" containerID="560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb"
Sep 05 03:26:51.378258 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.378211    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb"} err="failed to get container status \"560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb\": rpc error: code = NotFound desc = could not find container \"560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb\": container with ID starting with 560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb not found: ID does not exist"
Sep 05 03:26:51.378258 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.378227    2557 scope.go:117] "RemoveContainer" containerID="0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909"
Sep 05 03:26:51.378510 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:26:51.378484    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909\": container with ID starting with 0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909 not found: ID does not exist" containerID="0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909"
Sep 05 03:26:51.378609 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.378514    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909"} err="failed to get container status \"0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909\": rpc error: code = NotFound desc = could not find container \"0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909\": container with ID starting with 0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909 not found: ID does not exist"
Sep 05 03:26:51.378609 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.378529    2557 scope.go:117] "RemoveContainer" containerID="efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2"
Sep 05 03:26:51.378789 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.378770    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2"} err="failed to get container status \"efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2\": rpc error: code = NotFound desc = could not find container \"efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2\": container with ID starting with efc84ad203c6cb40882a02140755cecd185b65d4f6cc83729f361ca66d89a0b2 not found: ID does not exist"
Sep 05 03:26:51.378854 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.378792    2557 scope.go:117] "RemoveContainer" containerID="560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb"
Sep 05 03:26:51.379063 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.379040    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb"} err="failed to get container status \"560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb\": rpc error: code = NotFound desc = could not find container \"560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb\": container with ID starting with 560f0c36be150b87d533b31be6683f168ff11c8b940442245b3bb2d3a42208bb not found: ID does not exist"
Sep 05 03:26:51.379121 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.379064    2557 scope.go:117] "RemoveContainer" containerID="0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909"
Sep 05 03:26:51.379323 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:51.379296    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909"} err="failed to get container status \"0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909\": rpc error: code = NotFound desc = could not find container \"0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909\": container with ID starting with 0904aef8082a991c66492662734efe5991fc582beac13e357db1bbdc4d9c6909 not found: ID does not exist"
Sep 05 03:26:52.991475 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:52.991421    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" path="/var/lib/kubelet/pods/c155ba4f-db57-4c9a-a66d-b82c74347162/volumes"
Sep 05 03:26:55.137350 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:55.137295    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused"
Sep 05 03:26:57.018371 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:26:57.018329    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:27:05.137663 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:05.137615    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused"
Sep 05 03:27:15.137411 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:15.137310    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused"
Sep 05 03:27:23.027888 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.027836    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:27:23.217267 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.217229    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"]
Sep 05 03:27:23.217831 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.217808    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main"
Sep 05 03:27:23.217888 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.217836    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main"
Sep 05 03:27:23.217888 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.217852    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="storage-initializer"
Sep 05 03:27:23.217888 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.217862    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="storage-initializer"
Sep 05 03:27:23.217888 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.217877    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="llm-d-routing-sidecar"
Sep 05 03:27:23.217888 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.217888    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="llm-d-routing-sidecar"
Sep 05 03:27:23.218048 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.218000    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="main"
Sep 05 03:27:23.218048 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.218016    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c155ba4f-db57-4c9a-a66d-b82c74347162" containerName="llm-d-routing-sidecar"
Sep 05 03:27:23.222870 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.222843    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.225110 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.225083    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 05 03:27:23.225231 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.225179    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-ppzxb\""
Sep 05 03:27:23.225302 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.225232    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 05 03:27:23.225370 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.225354    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 05 03:27:23.235628 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.235600    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"]
Sep 05 03:27:23.378385 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.378350    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.378588 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.378392    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.378588 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.378415    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcw8g\" (UniqueName: \"kubernetes.io/projected/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-kube-api-access-kcw8g\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.378588 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.378485    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.378588 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.378515    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.378588 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.378533    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.378837 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.378637    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.378837 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.378758    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.378837 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.378799    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.479996 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.479954    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.480176 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.480047    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.480176 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.480078    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.480176 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.480121    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.480176 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.480158    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.480413 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.480191    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcw8g\" (UniqueName: \"kubernetes.io/projected/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-kube-api-access-kcw8g\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.480413 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.480223    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.480413 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.480248    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.480413 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.480273    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.480648 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.480519    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.480648 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.480596    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.480827 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.480795    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.480952 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.480834    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.480952 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.480854    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.482666 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.482640    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.482887 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.482859    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.488776 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.488752    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.488939 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.488918    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcw8g\" (UniqueName: \"kubernetes.io/projected/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-kube-api-access-kcw8g\") pod \"router-gateway-2-openshift-default-5c874f788-dk5dn\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.536126 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.536098    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:23.844796 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.844756    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"]
Sep 05 03:27:23.847253 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:27:23.847226    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b133aaf_3bd1_443d_8cd5_9bbc11de15c0.slice/crio-35df5c685ce8468fc9af80331b50aa6df5b5e69bb3888feedaf701af9fddef95 WatchSource:0}: Error finding container 35df5c685ce8468fc9af80331b50aa6df5b5e69bb3888feedaf701af9fddef95: Status 404 returned error can't find the container with id 35df5c685ce8468fc9af80331b50aa6df5b5e69bb3888feedaf701af9fddef95
Sep 05 03:27:23.849607 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.849567    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:27:23.849677 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.849656    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:27:23.849713 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:23.849697    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:27:24.286623 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:24.286511    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:27:24.531893 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:24.531853    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn" event={"ID":"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0","Type":"ContainerStarted","Data":"796a788fdd7ec93c37acfcc47637cf35d69f71797d733449fb9307d806264bb5"}
Sep 05 03:27:24.532072 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:24.531912    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn" event={"ID":"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0","Type":"ContainerStarted","Data":"35df5c685ce8468fc9af80331b50aa6df5b5e69bb3888feedaf701af9fddef95"}
Sep 05 03:27:24.536297 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:24.536273    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:24.541916 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:24.541837    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:24.552255 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:24.552208    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn" podStartSLOduration=1.552193575 podStartE2EDuration="1.552193575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:27:24.551476984 +0000 UTC m=+4148.106633295" watchObservedRunningTime="2026-09-05 03:27:24.552193575 +0000 UTC m=+4148.107349884"
Sep 05 03:27:25.137393 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:25.137351    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused"
Sep 05 03:27:25.537298 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:25.537212    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:25.540173 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:25.540145    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:26.676603 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:26.676297    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:27:29.748352 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:29.748314    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:27:35.137277 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:35.137238    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused"
Sep 05 03:27:35.405735 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:35.405693    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:27:35.469734 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:35.469688    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:27:35.610906 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:35.610857    2557 generic.go:356] "Generic (PLEG): container finished" podID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" containerID="796a788fdd7ec93c37acfcc47637cf35d69f71797d733449fb9307d806264bb5" exitCode=0
Sep 05 03:27:35.611088 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:35.610928    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn" event={"ID":"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0","Type":"ContainerDied","Data":"796a788fdd7ec93c37acfcc47637cf35d69f71797d733449fb9307d806264bb5"}
Sep 05 03:27:35.611354 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:35.611339    2557 scope.go:117] "RemoveContainer" containerID="796a788fdd7ec93c37acfcc47637cf35d69f71797d733449fb9307d806264bb5"
Sep 05 03:27:35.612330 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:35.612299    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:27:35.612408 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:35.612370    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:27:35.612481 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:35.612406    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 05 03:27:36.619265 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:36.619223    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn" event={"ID":"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0","Type":"ContainerStarted","Data":"c2a9c251767284db85c4f36ac5efcc1586c874c7d617f244066cda9644c6fe82"}
Sep 05 03:27:37.536354 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:37.536315    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:37.537665 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:37.537621    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn" podUID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Sep 05 03:27:38.536584 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:38.536540    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn" podUID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Sep 05 03:27:39.536860 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:39.536814    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn" podUID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Sep 05 03:27:40.537353 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:40.537308    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn" podUID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Sep 05 03:27:40.537870 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:40.537754    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:41.536620 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:41.536576    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn" podUID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Sep 05 03:27:42.541161 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:42.541112    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:42.542304 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:42.542278    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:27:45.137538 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:45.137490    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused"
Sep 05 03:27:51.455515 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.455469    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"]
Sep 05 03:27:51.459710 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.459683    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.462243 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.462214    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-xjt9s\""
Sep 05 03:27:51.462373 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.462356    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:27:51.462484 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.462224    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 05 03:27:51.462569 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.462245    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-27csp\""
Sep 05 03:27:51.462569 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.462538    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 05 03:27:51.474757 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.474729    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"]
Sep 05 03:27:51.509565 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.509526    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:27:51.561421 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.561380    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d050c692-f598-432e-9ccd-506344152e00-tls-certs\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.561421 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.561426    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-home\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.561645 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.561499    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-dshm\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.561645 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.561561    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.561645 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.561608    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-model-cache\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.561800 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.561652    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d050c692-f598-432e-9ccd-506344152e00-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.561800 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.561730    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-tmp-dir\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.561897 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.561805    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k6wc\" (UniqueName: \"kubernetes.io/projected/d050c692-f598-432e-9ccd-506344152e00-kube-api-access-7k6wc\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.662468 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.662419    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-tmp-dir\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.662657 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.662504    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k6wc\" (UniqueName: \"kubernetes.io/projected/d050c692-f598-432e-9ccd-506344152e00-kube-api-access-7k6wc\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.662657 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.662534    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d050c692-f598-432e-9ccd-506344152e00-tls-certs\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.662657 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.662555    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-home\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.662657 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.662583    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-dshm\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.662657 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.662617    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.662657 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.662653    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-model-cache\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.663005 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.662693    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d050c692-f598-432e-9ccd-506344152e00-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.663066 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.663013    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-tmp-dir\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.663212 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.663177    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.663365 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.663349    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-home\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.663451 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.663410    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-model-cache\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.663566 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.663543    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d050c692-f598-432e-9ccd-506344152e00-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.665205 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.665188    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d050c692-f598-432e-9ccd-506344152e00-tls-certs\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.665518 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.665499    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-dshm\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.669871 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.669843    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k6wc\" (UniqueName: \"kubernetes.io/projected/d050c692-f598-432e-9ccd-506344152e00-kube-api-access-7k6wc\") pod \"router-with-refs-pd-test-kserve-588ffb4c9-smddc\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:51.772780 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:51.772681    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:52.005013 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:52.004977    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:27:52.055730 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:52.053616    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"]
Sep 05 03:27:52.058354 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:27:52.058092    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd050c692_f598_432e_9ccd_506344152e00.slice/crio-25138024598cbb6e0555e6d59ec6814e4273d0224d0cdf6dac19df17d99e3ffd WatchSource:0}: Error finding container 25138024598cbb6e0555e6d59ec6814e4273d0224d0cdf6dac19df17d99e3ffd: Status 404 returned error can't find the container with id 25138024598cbb6e0555e6d59ec6814e4273d0224d0cdf6dac19df17d99e3ffd
Sep 05 03:27:52.147234 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:52.147193    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:27:52.738422 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:52.738371    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" event={"ID":"d050c692-f598-432e-9ccd-506344152e00","Type":"ContainerStarted","Data":"5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0"}
Sep 05 03:27:52.738891 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:52.738464    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" event={"ID":"d050c692-f598-432e-9ccd-506344152e00","Type":"ContainerStarted","Data":"25138024598cbb6e0555e6d59ec6814e4273d0224d0cdf6dac19df17d99e3ffd"}
Sep 05 03:27:52.738891 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:52.738631    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:27:53.750836 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:53.750788    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" event={"ID":"d050c692-f598-432e-9ccd-506344152e00","Type":"ContainerStarted","Data":"373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee"}
Sep 05 03:27:55.137757 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:55.137708    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused"
Sep 05 03:27:55.766320 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:55.766273    2557 generic.go:356] "Generic (PLEG): container finished" podID="d050c692-f598-432e-9ccd-506344152e00" containerID="373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee" exitCode=0
Sep 05 03:27:55.766549 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:55.766348    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" event={"ID":"d050c692-f598-432e-9ccd-506344152e00","Type":"ContainerDied","Data":"373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee"}
Sep 05 03:27:56.776988 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:56.776942    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" event={"ID":"d050c692-f598-432e-9ccd-506344152e00","Type":"ContainerStarted","Data":"42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af"}
Sep 05 03:27:56.802210 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:27:56.802142    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" podStartSLOduration=3.355423942 podStartE2EDuration="5.802118736s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.446694794s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:27:56.798284207 +0000 UTC m=+4180.353440516" watchObservedRunningTime="2026-09-05 03:27:56.802118736 +0000 UTC m=+4180.357275047"
Sep 05 03:28:01.773418 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:01.773374    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:28:01.773871 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:01.773454    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:28:01.775023 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:01.774992    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8001/health\": dial tcp 10.133.0.105:8001: connect: connection refused"
Sep 05 03:28:05.137373 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:05.137327    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": dial tcp 10.133.0.103:8000: connect: connection refused"
Sep 05 03:28:11.773725 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:11.773658    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8001/health\": dial tcp 10.133.0.105:8001: connect: connection refused"
Sep 05 03:28:11.793503 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:11.793466    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:28:15.148317 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:15.148278    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:28:15.156514 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:15.156483    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:28:17.376078 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:17.376043    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:28:17.388380 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:17.388351    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:28:20.334972 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:20.334936    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:28:20.343139 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:20.343104    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"]
Sep 05 03:28:20.428837 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:20.428801    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:28:21.773819 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:21.773761    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8001/health\": dial tcp 10.133.0.105:8001: connect: connection refused"
Sep 05 03:28:31.773909 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:31.773853    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8001/health\": dial tcp 10.133.0.105:8001: connect: connection refused"
Sep 05 03:28:35.369578 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:35.369529    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main" containerID="cri-o://5673f2712c00b39d13ed345a532eff15d13d1d9f8145db94ef704c265d3e182c" gracePeriod=45
Sep 05 03:28:36.015375 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.015337    2557 generic.go:356] "Generic (PLEG): container finished" podID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerID="5673f2712c00b39d13ed345a532eff15d13d1d9f8145db94ef704c265d3e182c" exitCode=0
Sep 05 03:28:36.015565 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.015398    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" event={"ID":"b578de5c-386f-4b91-a464-5ccd113fc4eb","Type":"ContainerDied","Data":"5673f2712c00b39d13ed345a532eff15d13d1d9f8145db94ef704c265d3e182c"}
Sep 05 03:28:36.141705 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.141677    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:28:36.201746 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.201644    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b578de5c-386f-4b91-a464-5ccd113fc4eb-tls-certs\") pod \"b578de5c-386f-4b91-a464-5ccd113fc4eb\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") "
Sep 05 03:28:36.201746 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.201691    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b578de5c-386f-4b91-a464-5ccd113fc4eb-cabundle-cert\") pod \"b578de5c-386f-4b91-a464-5ccd113fc4eb\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") "
Sep 05 03:28:36.201746 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.201736    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-dshm\") pod \"b578de5c-386f-4b91-a464-5ccd113fc4eb\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") "
Sep 05 03:28:36.202039 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.201802    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-home\") pod \"b578de5c-386f-4b91-a464-5ccd113fc4eb\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") "
Sep 05 03:28:36.202039 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.201839    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-kserve-provision-location\") pod \"b578de5c-386f-4b91-a464-5ccd113fc4eb\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") "
Sep 05 03:28:36.202039 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.201861    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-model-cache\") pod \"b578de5c-386f-4b91-a464-5ccd113fc4eb\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") "
Sep 05 03:28:36.202039 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.201893    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-tmp-dir\") pod \"b578de5c-386f-4b91-a464-5ccd113fc4eb\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") "
Sep 05 03:28:36.202039 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.201926    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jpnr\" (UniqueName: \"kubernetes.io/projected/b578de5c-386f-4b91-a464-5ccd113fc4eb-kube-api-access-2jpnr\") pod \"b578de5c-386f-4b91-a464-5ccd113fc4eb\" (UID: \"b578de5c-386f-4b91-a464-5ccd113fc4eb\") "
Sep 05 03:28:36.202307 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.202154    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b578de5c-386f-4b91-a464-5ccd113fc4eb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b578de5c-386f-4b91-a464-5ccd113fc4eb" (UID: "b578de5c-386f-4b91-a464-5ccd113fc4eb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:28:36.202778 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.202747    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-home" (OuterVolumeSpecName: "home") pod "b578de5c-386f-4b91-a464-5ccd113fc4eb" (UID: "b578de5c-386f-4b91-a464-5ccd113fc4eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:28:36.203009 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.202987    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:28:36.203205 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.203188    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b578de5c-386f-4b91-a464-5ccd113fc4eb-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:28:36.204343 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.204315    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "b578de5c-386f-4b91-a464-5ccd113fc4eb" (UID: "b578de5c-386f-4b91-a464-5ccd113fc4eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:28:36.204714 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.204684    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-dshm" (OuterVolumeSpecName: "dshm") pod "b578de5c-386f-4b91-a464-5ccd113fc4eb" (UID: "b578de5c-386f-4b91-a464-5ccd113fc4eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:28:36.204785 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.204684    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b578de5c-386f-4b91-a464-5ccd113fc4eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b578de5c-386f-4b91-a464-5ccd113fc4eb" (UID: "b578de5c-386f-4b91-a464-5ccd113fc4eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:28:36.204851 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.204818    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b578de5c-386f-4b91-a464-5ccd113fc4eb-kube-api-access-2jpnr" (OuterVolumeSpecName: "kube-api-access-2jpnr") pod "b578de5c-386f-4b91-a464-5ccd113fc4eb" (UID: "b578de5c-386f-4b91-a464-5ccd113fc4eb"). InnerVolumeSpecName "kube-api-access-2jpnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:28:36.220950 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.220895    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b578de5c-386f-4b91-a464-5ccd113fc4eb" (UID: "b578de5c-386f-4b91-a464-5ccd113fc4eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:28:36.234145 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.234108    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b578de5c-386f-4b91-a464-5ccd113fc4eb" (UID: "b578de5c-386f-4b91-a464-5ccd113fc4eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:28:36.304663 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.304623    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:28:36.304663 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.304658    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:28:36.304663 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.304671    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:28:36.304910 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.304685    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jpnr\" (UniqueName: \"kubernetes.io/projected/b578de5c-386f-4b91-a464-5ccd113fc4eb-kube-api-access-2jpnr\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:28:36.304910 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.304697    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b578de5c-386f-4b91-a464-5ccd113fc4eb-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:28:36.304910 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:36.304706    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b578de5c-386f-4b91-a464-5ccd113fc4eb-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:28:37.022719 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:37.022680    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l" event={"ID":"b578de5c-386f-4b91-a464-5ccd113fc4eb","Type":"ContainerDied","Data":"1df58cff2cb1245690371fad6a549e5b328f0f49f84ffd6418b818ffdac3b5f9"}
Sep 05 03:28:37.023108 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:37.022725    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"
Sep 05 03:28:37.023108 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:37.022757    2557 scope.go:117] "RemoveContainer" containerID="5673f2712c00b39d13ed345a532eff15d13d1d9f8145db94ef704c265d3e182c"
Sep 05 03:28:37.032469 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:37.032428    2557 scope.go:117] "RemoveContainer" containerID="4bee910528a50073c9b37a4b6225ca41341058f7883b7710004a35c9489f41b8"
Sep 05 03:28:37.043562 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:37.043532    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"]
Sep 05 03:28:37.047332 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:37.047305    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8f4cd5b9-db72l"]
Sep 05 03:28:38.991867 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:38.991777    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" path="/var/lib/kubelet/pods/b578de5c-386f-4b91-a464-5ccd113fc4eb/volumes"
Sep 05 03:28:41.773376 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:41.773322    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8001/health\": dial tcp 10.133.0.105:8001: connect: connection refused"
Sep 05 03:28:42.698743 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:42.698694    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:28:51.773400 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:28:51.773349    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8001/health\": dial tcp 10.133.0.105:8001: connect: connection refused"
Sep 05 03:29:01.773463 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:01.773386    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8001/health\": dial tcp 10.133.0.105:8001: connect: connection refused"
Sep 05 03:29:10.453150 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:10.453112    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:29:10.552401 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:10.552361    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:29:11.773851 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:11.773809    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8001/health\": dial tcp 10.133.0.105:8001: connect: connection refused"
Sep 05 03:29:16.461097 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:16.461053    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:29:21.773428 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:21.773378    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8001/health\": dial tcp 10.133.0.105:8001: connect: connection refused"
Sep 05 03:29:21.901880 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:21.901840    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:29:21.963953 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:21.963916    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:29:31.773754 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:31.773694    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8001/health\": dial tcp 10.133.0.105:8001: connect: connection refused"
Sep 05 03:29:32.140962 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:32.140915    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:29:36.214391 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.214354    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"]
Sep 05 03:29:36.214836 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.214747    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main"
Sep 05 03:29:36.214836 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.214762    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main"
Sep 05 03:29:36.214836 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.214774    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="storage-initializer"
Sep 05 03:29:36.214836 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.214780    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="storage-initializer"
Sep 05 03:29:36.214995 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.214847    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b578de5c-386f-4b91-a464-5ccd113fc4eb" containerName="main"
Sep 05 03:29:36.217874 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.217853    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.220531 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.220502    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 05 03:29:36.220531 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.220522    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 05 03:29:36.220787 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.220738    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 05 03:29:36.220787 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.220773    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 03:29:36.220917 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.220852    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-8g5ll\""
Sep 05 03:29:36.220917 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.220891    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 05 03:29:36.228579 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.228552    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"]
Sep 05 03:29:36.238335 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.238295    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:29:36.349238 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.349197    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-kserve-provision-location\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.349419 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.349253    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44wfv\" (UniqueName: \"kubernetes.io/projected/838d15a6-b734-4b5d-9cb5-1676591f0d33-kube-api-access-44wfv\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.349419 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.349293    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-model-cache\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.349560 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.349409    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-tmp-dir\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.349560 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.349528    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838d15a6-b734-4b5d-9cb5-1676591f0d33-cabundle-cert\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.349648 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.349568    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-home\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.349648 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.349599    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-dshm\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.349758 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.349672    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838d15a6-b734-4b5d-9cb5-1676591f0d33-tls-certs\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.445294 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.445255    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:29:36.451205 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.451173    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838d15a6-b734-4b5d-9cb5-1676591f0d33-tls-certs\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.451356 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.451240    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-kserve-provision-location\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.451356 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.451272    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44wfv\" (UniqueName: \"kubernetes.io/projected/838d15a6-b734-4b5d-9cb5-1676591f0d33-kube-api-access-44wfv\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.451356 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.451299    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-model-cache\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.451356 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.451339    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-tmp-dir\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.451603 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.451366    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838d15a6-b734-4b5d-9cb5-1676591f0d33-cabundle-cert\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.451603 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.451386    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-home\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.451603 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.451409    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-dshm\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.451784 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.451754    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-kserve-provision-location\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.451828 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.451782    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-model-cache\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.452123 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.452092    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-home\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.452203 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.452124    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-tmp-dir\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.452203 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.452173    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838d15a6-b734-4b5d-9cb5-1676591f0d33-cabundle-cert\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.454564 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.454536    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838d15a6-b734-4b5d-9cb5-1676591f0d33-tls-certs\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.454693 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.454675    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-dshm\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.460406 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.460380    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44wfv\" (UniqueName: \"kubernetes.io/projected/838d15a6-b734-4b5d-9cb5-1676591f0d33-kube-api-access-44wfv\") pod \"prestop-hook-test-kserve-7787c46b47-mnhl2\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.530101 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.530013    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:36.817932 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:36.817901    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"]
Sep 05 03:29:36.819622 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:29:36.819592    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838d15a6_b734_4b5d_9cb5_1676591f0d33.slice/crio-cb1c2d93e1b53711d1c10a230e7382f8b2338ccd53cbdd5c71c35995ed46fb29 WatchSource:0}: Error finding container cb1c2d93e1b53711d1c10a230e7382f8b2338ccd53cbdd5c71c35995ed46fb29: Status 404 returned error can't find the container with id cb1c2d93e1b53711d1c10a230e7382f8b2338ccd53cbdd5c71c35995ed46fb29
Sep 05 03:29:37.372056 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:37.372004    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" event={"ID":"838d15a6-b734-4b5d-9cb5-1676591f0d33","Type":"ContainerStarted","Data":"5d2f7abc02b81bbaf5adb83a2876746efc0591f0eff6e9a0ba8f91ae7c50c0f0"}
Sep 05 03:29:37.372575 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:37.372084    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" event={"ID":"838d15a6-b734-4b5d-9cb5-1676591f0d33","Type":"ContainerStarted","Data":"cb1c2d93e1b53711d1c10a230e7382f8b2338ccd53cbdd5c71c35995ed46fb29"}
Sep 05 03:29:40.396460 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:40.396395    2557 generic.go:356] "Generic (PLEG): container finished" podID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerID="5d2f7abc02b81bbaf5adb83a2876746efc0591f0eff6e9a0ba8f91ae7c50c0f0" exitCode=0
Sep 05 03:29:40.396854 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:40.396467    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" event={"ID":"838d15a6-b734-4b5d-9cb5-1676591f0d33","Type":"ContainerDied","Data":"5d2f7abc02b81bbaf5adb83a2876746efc0591f0eff6e9a0ba8f91ae7c50c0f0"}
Sep 05 03:29:41.405925 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:41.405882    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" event={"ID":"838d15a6-b734-4b5d-9cb5-1676591f0d33","Type":"ContainerStarted","Data":"a7006a05fde8c03d9d6c3df766bfefa28b1ee1a70893f58df94b1e3b59ace7b8"}
Sep 05 03:29:41.428563 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:41.428488    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" podStartSLOduration=2.617308716 podStartE2EDuration="5.428472261s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.811163545s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:29:41.425276412 +0000 UTC m=+4284.980432720" watchObservedRunningTime="2026-09-05 03:29:41.428472261 +0000 UTC m=+4284.983628569"
Sep 05 03:29:41.530426 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:41.530387    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:41.773552 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:41.773421    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8001/health\": dial tcp 10.133.0.105:8001: connect: connection refused"
Sep 05 03:29:46.531011 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:46.530963    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:29:46.532783 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:46.532742    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": dial tcp 10.133.0.106:8000: connect: connection refused"
Sep 05 03:29:51.783664 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:51.783627    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:29:51.802750 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:51.802714    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:29:53.412191 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:53.412147    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:29:56.531486 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:29:56.531408    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": dial tcp 10.133.0.106:8000: connect: connection refused"
Sep 05 03:30:04.661725 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:04.661683    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"]
Sep 05 03:30:04.936313 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:04.936210    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:30:05.036912 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:05.036862    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:30:06.531586 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:06.531532    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": dial tcp 10.133.0.106:8000: connect: connection refused"
Sep 05 03:30:16.530722 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:16.530614    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": dial tcp 10.133.0.106:8000: connect: connection refused"
Sep 05 03:30:19.687447 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:19.687390    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main" containerID="cri-o://42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af" gracePeriod=45
Sep 05 03:30:20.305033 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.304983    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="llm-d-routing-sidecar" containerID="cri-o://5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0" gracePeriod=45
Sep 05 03:30:20.558766 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.558737    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:30:20.631136 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.631099    2557 generic.go:356] "Generic (PLEG): container finished" podID="d050c692-f598-432e-9ccd-506344152e00" containerID="42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af" exitCode=0
Sep 05 03:30:20.631136 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.631127    2557 generic.go:356] "Generic (PLEG): container finished" podID="d050c692-f598-432e-9ccd-506344152e00" containerID="5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0" exitCode=0
Sep 05 03:30:20.631361 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.631162    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"
Sep 05 03:30:20.631361 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.631175    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" event={"ID":"d050c692-f598-432e-9ccd-506344152e00","Type":"ContainerDied","Data":"42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af"}
Sep 05 03:30:20.631361 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.631224    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" event={"ID":"d050c692-f598-432e-9ccd-506344152e00","Type":"ContainerDied","Data":"5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0"}
Sep 05 03:30:20.631361 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.631242    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc" event={"ID":"d050c692-f598-432e-9ccd-506344152e00","Type":"ContainerDied","Data":"25138024598cbb6e0555e6d59ec6814e4273d0224d0cdf6dac19df17d99e3ffd"}
Sep 05 03:30:20.631361 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.631262    2557 scope.go:117] "RemoveContainer" containerID="42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af"
Sep 05 03:30:20.640954 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.640927    2557 scope.go:117] "RemoveContainer" containerID="373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee"
Sep 05 03:30:20.649757 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.649735    2557 scope.go:117] "RemoveContainer" containerID="5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0"
Sep 05 03:30:20.657844 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.657819    2557 scope.go:117] "RemoveContainer" containerID="42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af"
Sep 05 03:30:20.658120 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:30:20.658103    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af\": container with ID starting with 42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af not found: ID does not exist" containerID="42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af"
Sep 05 03:30:20.658174 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.658130    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af"} err="failed to get container status \"42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af\": rpc error: code = NotFound desc = could not find container \"42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af\": container with ID starting with 42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af not found: ID does not exist"
Sep 05 03:30:20.658174 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.658147    2557 scope.go:117] "RemoveContainer" containerID="373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee"
Sep 05 03:30:20.658431 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:30:20.658411    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee\": container with ID starting with 373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee not found: ID does not exist" containerID="373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee"
Sep 05 03:30:20.658517 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.658456    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee"} err="failed to get container status \"373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee\": rpc error: code = NotFound desc = could not find container \"373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee\": container with ID starting with 373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee not found: ID does not exist"
Sep 05 03:30:20.658517 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.658474    2557 scope.go:117] "RemoveContainer" containerID="5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0"
Sep 05 03:30:20.658758 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:30:20.658735    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0\": container with ID starting with 5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0 not found: ID does not exist" containerID="5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0"
Sep 05 03:30:20.658841 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.658767    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0"} err="failed to get container status \"5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0\": rpc error: code = NotFound desc = could not find container \"5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0\": container with ID starting with 5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0 not found: ID does not exist"
Sep 05 03:30:20.658841 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.658788    2557 scope.go:117] "RemoveContainer" containerID="42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af"
Sep 05 03:30:20.659039 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.659018    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af"} err="failed to get container status \"42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af\": rpc error: code = NotFound desc = could not find container \"42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af\": container with ID starting with 42afddc00d410245312592f2c387d2082ad6788429fd05b650ca6946217eb7af not found: ID does not exist"
Sep 05 03:30:20.659111 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.659041    2557 scope.go:117] "RemoveContainer" containerID="373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee"
Sep 05 03:30:20.659285 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.659268    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee"} err="failed to get container status \"373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee\": rpc error: code = NotFound desc = could not find container \"373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee\": container with ID starting with 373ad8e7cb46b1dec1dd9e8b33ecef55ef0b27236094d45dec34f2cb656e9eee not found: ID does not exist"
Sep 05 03:30:20.659346 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.659285    2557 scope.go:117] "RemoveContainer" containerID="5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0"
Sep 05 03:30:20.659551 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.659524    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0"} err="failed to get container status \"5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0\": rpc error: code = NotFound desc = could not find container \"5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0\": container with ID starting with 5fe4fe527a76a758529a4e858d0b6806e15fe5dd23519235587d42c92cf6e9a0 not found: ID does not exist"
Sep 05 03:30:20.669927 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.669910    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-home\") pod \"d050c692-f598-432e-9ccd-506344152e00\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") "
Sep 05 03:30:20.669993 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.669935    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-kserve-provision-location\") pod \"d050c692-f598-432e-9ccd-506344152e00\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") "
Sep 05 03:30:20.669993 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.669987    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d050c692-f598-432e-9ccd-506344152e00-cabundle-cert\") pod \"d050c692-f598-432e-9ccd-506344152e00\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") "
Sep 05 03:30:20.670083 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.670007    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-dshm\") pod \"d050c692-f598-432e-9ccd-506344152e00\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") "
Sep 05 03:30:20.670083 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.670062    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-model-cache\") pod \"d050c692-f598-432e-9ccd-506344152e00\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") "
Sep 05 03:30:20.670194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.670093    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k6wc\" (UniqueName: \"kubernetes.io/projected/d050c692-f598-432e-9ccd-506344152e00-kube-api-access-7k6wc\") pod \"d050c692-f598-432e-9ccd-506344152e00\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") "
Sep 05 03:30:20.670194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.670151    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d050c692-f598-432e-9ccd-506344152e00-tls-certs\") pod \"d050c692-f598-432e-9ccd-506344152e00\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") "
Sep 05 03:30:20.670310 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.670214    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-tmp-dir\") pod \"d050c692-f598-432e-9ccd-506344152e00\" (UID: \"d050c692-f598-432e-9ccd-506344152e00\") "
Sep 05 03:30:20.670609 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.670564    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d050c692-f598-432e-9ccd-506344152e00-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d050c692-f598-432e-9ccd-506344152e00" (UID: "d050c692-f598-432e-9ccd-506344152e00"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:30:20.670609 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.670583    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-home" (OuterVolumeSpecName: "home") pod "d050c692-f598-432e-9ccd-506344152e00" (UID: "d050c692-f598-432e-9ccd-506344152e00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:30:20.670952 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.670720    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-model-cache" (OuterVolumeSpecName: "model-cache") pod "d050c692-f598-432e-9ccd-506344152e00" (UID: "d050c692-f598-432e-9ccd-506344152e00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:30:20.672602 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.672578    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d050c692-f598-432e-9ccd-506344152e00-kube-api-access-7k6wc" (OuterVolumeSpecName: "kube-api-access-7k6wc") pod "d050c692-f598-432e-9ccd-506344152e00" (UID: "d050c692-f598-432e-9ccd-506344152e00"). InnerVolumeSpecName "kube-api-access-7k6wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:30:20.672695 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.672617    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d050c692-f598-432e-9ccd-506344152e00-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d050c692-f598-432e-9ccd-506344152e00" (UID: "d050c692-f598-432e-9ccd-506344152e00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:30:20.672780 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.672762    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-dshm" (OuterVolumeSpecName: "dshm") pod "d050c692-f598-432e-9ccd-506344152e00" (UID: "d050c692-f598-432e-9ccd-506344152e00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:30:20.690898 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.690863    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d050c692-f598-432e-9ccd-506344152e00" (UID: "d050c692-f598-432e-9ccd-506344152e00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:30:20.699943 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.699915    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d050c692-f598-432e-9ccd-506344152e00" (UID: "d050c692-f598-432e-9ccd-506344152e00"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:30:20.771044 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.771016    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.771044 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.771044    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.771044 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.771056    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k6wc\" (UniqueName: \"kubernetes.io/projected/d050c692-f598-432e-9ccd-506344152e00-kube-api-access-7k6wc\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.771276 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.771065    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d050c692-f598-432e-9ccd-506344152e00-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.771276 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.771075    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.771276 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.771082    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.771276 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.771090    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d050c692-f598-432e-9ccd-506344152e00-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.771276 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.771099    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d050c692-f598-432e-9ccd-506344152e00-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:30:20.955601 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.955564    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"]
Sep 05 03:30:20.961375 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:20.961348    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-588ffb4c9-smddc"]
Sep 05 03:30:21.001134 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:21.001097    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d050c692-f598-432e-9ccd-506344152e00" path="/var/lib/kubelet/pods/d050c692-f598-432e-9ccd-506344152e00/volumes"
Sep 05 03:30:26.530618 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:26.530564    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": dial tcp 10.133.0.106:8000: connect: connection refused"
Sep 05 03:30:36.531125 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:36.531074    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": dial tcp 10.133.0.106:8000: connect: connection refused"
Sep 05 03:30:46.530948 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:46.530893    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": dial tcp 10.133.0.106:8000: connect: connection refused"
Sep 05 03:30:54.821171 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:54.821125    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:30:54.912771 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:54.912730    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:30:56.530830 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:56.530786    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": dial tcp 10.133.0.106:8000: connect: connection refused"
Sep 05 03:30:59.763947 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:59.763904    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"]
Sep 05 03:30:59.764370 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:59.764143    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn" podUID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" containerName="istio-proxy" containerID="cri-o://c2a9c251767284db85c4f36ac5efcc1586c874c7d617f244066cda9644c6fe82" gracePeriod=30
Sep 05 03:30:59.804396 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:30:59.804358    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:31:01.523771 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:01.523716    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:31:01.941615 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:01.941563    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:31:04.897264 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:04.897228    2557 generic.go:356] "Generic (PLEG): container finished" podID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" containerID="c2a9c251767284db85c4f36ac5efcc1586c874c7d617f244066cda9644c6fe82" exitCode=0
Sep 05 03:31:04.897692 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:04.897309    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn" event={"ID":"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0","Type":"ContainerDied","Data":"c2a9c251767284db85c4f36ac5efcc1586c874c7d617f244066cda9644c6fe82"}
Sep 05 03:31:04.897692 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:04.897375    2557 scope.go:117] "RemoveContainer" containerID="796a788fdd7ec93c37acfcc47637cf35d69f71797d733449fb9307d806264bb5"
Sep 05 03:31:05.027947 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.027852    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:31:05.081933 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.081892    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcw8g\" (UniqueName: \"kubernetes.io/projected/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-kube-api-access-kcw8g\") pod \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") "
Sep 05 03:31:05.082126 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.081951    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-credential-socket\") pod \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") "
Sep 05 03:31:05.082126 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.081985    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-podinfo\") pod \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") "
Sep 05 03:31:05.082238 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.082170    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istiod-ca-cert\") pod \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") "
Sep 05 03:31:05.082289 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.082267    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-data\") pod \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") "
Sep 05 03:31:05.082338 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.082293    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-workload-certs\") pod \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") "
Sep 05 03:31:05.082388 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.082363    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" (UID: "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:05.082715 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.082528    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" (UID: "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:05.082715 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.082574    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-workload-socket\") pod \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") "
Sep 05 03:31:05.082715 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.082600    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" (UID: "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:05.082715 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.082618    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-token\") pod \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") "
Sep 05 03:31:05.082715 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.082654    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-envoy\") pod \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\" (UID: \"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0\") "
Sep 05 03:31:05.083012 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.082779    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" (UID: "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:05.083012 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.082980    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-credential-socket\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:05.083012 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.083000    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-data\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:05.083150 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.083017    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-workload-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:05.083150 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.083032    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-workload-socket\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:05.083481 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.083410    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" (UID: "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:31:05.084706 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.084679    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" (UID: "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 03:31:05.084800 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.084702    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-kube-api-access-kcw8g" (OuterVolumeSpecName: "kube-api-access-kcw8g") pod "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" (UID: "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0"). InnerVolumeSpecName "kube-api-access-kcw8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:31:05.085654 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.085629    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" (UID: "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:31:05.085755 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.085665    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" (UID: "6b133aaf-3bd1-443d-8cd5-9bbc11de15c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:05.184258 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.184210    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcw8g\" (UniqueName: \"kubernetes.io/projected/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-kube-api-access-kcw8g\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:05.184258 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.184250    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-podinfo\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:05.184258 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.184263    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istiod-ca-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:05.184515 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.184277    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-token\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:05.184515 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.184289    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0-istio-envoy\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:05.905003 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.904964    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"
Sep 05 03:31:05.905502 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.904969    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn" event={"ID":"6b133aaf-3bd1-443d-8cd5-9bbc11de15c0","Type":"ContainerDied","Data":"35df5c685ce8468fc9af80331b50aa6df5b5e69bb3888feedaf701af9fddef95"}
Sep 05 03:31:05.905502 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.905149    2557 scope.go:117] "RemoveContainer" containerID="c2a9c251767284db85c4f36ac5efcc1586c874c7d617f244066cda9644c6fe82"
Sep 05 03:31:05.951290 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.951238    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"]
Sep 05 03:31:05.956601 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:05.956557    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-dk5dn"]
Sep 05 03:31:06.530782 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:06.530725    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": dial tcp 10.133.0.106:8000: connect: connection refused"
Sep 05 03:31:06.991371 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:06.991334    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" path="/var/lib/kubelet/pods/6b133aaf-3bd1-443d-8cd5-9bbc11de15c0/volumes"
Sep 05 03:31:16.530779 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:16.530722    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": dial tcp 10.133.0.106:8000: connect: connection refused"
Sep 05 03:31:17.503647 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:17.503609    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:31:20.899676 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:20.899634    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:31:22.614242 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:22.614205    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:31:26.530940 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:26.530889    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": dial tcp 10.133.0.106:8000: connect: connection refused"
Sep 05 03:31:28.177553 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:28.177507    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:31:35.200190 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200137    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"]
Sep 05 03:31:35.200629 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200554    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" containerName="istio-proxy"
Sep 05 03:31:35.200629 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200570    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" containerName="istio-proxy"
Sep 05 03:31:35.200629 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200580    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="llm-d-routing-sidecar"
Sep 05 03:31:35.200629 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200586    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="llm-d-routing-sidecar"
Sep 05 03:31:35.200629 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200596    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" containerName="istio-proxy"
Sep 05 03:31:35.200629 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200603    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" containerName="istio-proxy"
Sep 05 03:31:35.200629 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200611    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main"
Sep 05 03:31:35.200629 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200617    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main"
Sep 05 03:31:35.200629 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200634    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="storage-initializer"
Sep 05 03:31:35.200950 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200640    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="storage-initializer"
Sep 05 03:31:35.200950 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200692    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="main"
Sep 05 03:31:35.200950 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200700    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d050c692-f598-432e-9ccd-506344152e00" containerName="llm-d-routing-sidecar"
Sep 05 03:31:35.200950 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200706    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" containerName="istio-proxy"
Sep 05 03:31:35.200950 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.200713    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b133aaf-3bd1-443d-8cd5-9bbc11de15c0" containerName="istio-proxy"
Sep 05 03:31:35.206636 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.206615    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.209756 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.209728    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-j5gxv\""
Sep 05 03:31:35.209903 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.209776    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 05 03:31:35.209903 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.209776    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 05 03:31:35.209903 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.209853    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 05 03:31:35.215709 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.215684    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"]
Sep 05 03:31:35.260927 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.260883    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.261121 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.260939    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.261121 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.261003    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg2fv\" (UniqueName: \"kubernetes.io/projected/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-kube-api-access-qg2fv\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.261121 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.261087    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.261297 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.261131    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.261297 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.261193    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.362316 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.362268    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.362316 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.362322    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.362619 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.362371    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.362619 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.362401    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.362619 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.362469    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.362619 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.362517    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg2fv\" (UniqueName: \"kubernetes.io/projected/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-kube-api-access-qg2fv\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.362860 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.362832    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.362958 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.362847    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.362958 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.362880    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.364668 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.364644    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.364869 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.364850    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.371108 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.371085    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg2fv\" (UniqueName: \"kubernetes.io/projected/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-kube-api-access-qg2fv\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-bkrsj\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.520159 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.520052    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:35.798607 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:35.798579    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"]
Sep 05 03:31:35.800867 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:31:35.800840    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84efea3_36c1_47a7_8e2f_c8afc0ce6e6b.slice/crio-3dd5332cc6eb9bc7643f64503ab9c1c18ea1f251f98447c2ccc1f053682f31b8 WatchSource:0}: Error finding container 3dd5332cc6eb9bc7643f64503ab9c1c18ea1f251f98447c2ccc1f053682f31b8: Status 404 returned error can't find the container with id 3dd5332cc6eb9bc7643f64503ab9c1c18ea1f251f98447c2ccc1f053682f31b8
Sep 05 03:31:36.081771 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:36.081730    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj" event={"ID":"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b","Type":"ContainerStarted","Data":"c1190795db903db2d35b9313428b7d7e95b57761ad820516f4a2f5d2c507a8a9"}
Sep 05 03:31:36.081943 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:36.081785    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj" event={"ID":"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b","Type":"ContainerStarted","Data":"3dd5332cc6eb9bc7643f64503ab9c1c18ea1f251f98447c2ccc1f053682f31b8"}
Sep 05 03:31:36.099718 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:36.099665    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj" podStartSLOduration=1.09964983 podStartE2EDuration="1.09964983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:31:36.099042933 +0000 UTC m=+4399.654199242" watchObservedRunningTime="2026-09-05 03:31:36.09964983 +0000 UTC m=+4399.654806139"
Sep 05 03:31:36.520327 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:36.520239    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:36.540631 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:36.540602    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:31:36.548659 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:36.548634    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:31:41.286325 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:41.286234    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"]
Sep 05 03:31:45.520873 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:45.520829    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:45.533699 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:45.533670    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:46.147801 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:46.147766    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:47.250515 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:47.250476    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:31:51.352051 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:51.352016    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:31:51.452369 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:51.452328    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:31:56.309371 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:56.309306    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main" containerID="cri-o://a7006a05fde8c03d9d6c3df766bfefa28b1ee1a70893f58df94b1e3b59ace7b8" gracePeriod=45
Sep 05 03:31:56.541366 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:56.541322    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/health\": dial tcp 10.133.0.106:8000: connect: connection refused"
Sep 05 03:31:56.906841 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:56.906799    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"]
Sep 05 03:31:56.908570 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:56.908535    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:31:56.917135 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:31:56.917097    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 05 03:31:56.917135 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:31:56.917135 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 03:31:56.917135 ip-10-0-143-128 kubenswrapper[2557]:  > containerID="c1190795db903db2d35b9313428b7d7e95b57761ad820516f4a2f5d2c507a8a9" cmd=["/bin/sleep","15"]
Sep 05 03:31:56.917479 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:56.917157    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj" message="" err=<
Sep 05 03:31:56.917479 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:31:56.917479 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 03:31:56.917479 ip-10-0-143-128 kubenswrapper[2557]:  >
Sep 05 03:31:56.917479 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:31:56.917174    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 05 03:31:56.917479 ip-10-0-143-128 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 05 03:31:56.917479 ip-10-0-143-128 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Sep 05 03:31:56.917479 ip-10-0-143-128 kubenswrapper[2557]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj" podUID="d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b" containerName="main" containerID="cri-o://c1190795db903db2d35b9313428b7d7e95b57761ad820516f4a2f5d2c507a8a9"
Sep 05 03:31:56.917479 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:56.917242    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj" podUID="d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b" containerName="main" containerID="cri-o://c1190795db903db2d35b9313428b7d7e95b57761ad820516f4a2f5d2c507a8a9" gracePeriod=60
Sep 05 03:31:56.971888 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:56.971839    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:31:57.093301 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.093242    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:31:57.137431 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.137387    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj" podUID="d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": dial tcp 10.133.0.107:8000: connect: connection refused"
Sep 05 03:31:57.161682 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.161652    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-model-cache\") pod \"838d15a6-b734-4b5d-9cb5-1676591f0d33\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") "
Sep 05 03:31:57.161858 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.161696    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838d15a6-b734-4b5d-9cb5-1676591f0d33-tls-certs\") pod \"838d15a6-b734-4b5d-9cb5-1676591f0d33\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") "
Sep 05 03:31:57.161858 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.161725    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838d15a6-b734-4b5d-9cb5-1676591f0d33-cabundle-cert\") pod \"838d15a6-b734-4b5d-9cb5-1676591f0d33\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") "
Sep 05 03:31:57.161858 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.161771    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44wfv\" (UniqueName: \"kubernetes.io/projected/838d15a6-b734-4b5d-9cb5-1676591f0d33-kube-api-access-44wfv\") pod \"838d15a6-b734-4b5d-9cb5-1676591f0d33\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") "
Sep 05 03:31:57.162035 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.161920    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-tmp-dir\") pod \"838d15a6-b734-4b5d-9cb5-1676591f0d33\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") "
Sep 05 03:31:57.162035 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.161968    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-dshm\") pod \"838d15a6-b734-4b5d-9cb5-1676591f0d33\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") "
Sep 05 03:31:57.162035 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.162014    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-kserve-provision-location\") pod \"838d15a6-b734-4b5d-9cb5-1676591f0d33\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") "
Sep 05 03:31:57.162194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.162063    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-home\") pod \"838d15a6-b734-4b5d-9cb5-1676591f0d33\" (UID: \"838d15a6-b734-4b5d-9cb5-1676591f0d33\") "
Sep 05 03:31:57.162194 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.161919    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-model-cache" (OuterVolumeSpecName: "model-cache") pod "838d15a6-b734-4b5d-9cb5-1676591f0d33" (UID: "838d15a6-b734-4b5d-9cb5-1676591f0d33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:57.162301 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.162202    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838d15a6-b734-4b5d-9cb5-1676591f0d33-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "838d15a6-b734-4b5d-9cb5-1676591f0d33" (UID: "838d15a6-b734-4b5d-9cb5-1676591f0d33"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 03:31:57.162461 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.162404    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:57.162461 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.162429    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838d15a6-b734-4b5d-9cb5-1676591f0d33-cabundle-cert\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:57.163173 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.163137    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-home" (OuterVolumeSpecName: "home") pod "838d15a6-b734-4b5d-9cb5-1676591f0d33" (UID: "838d15a6-b734-4b5d-9cb5-1676591f0d33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:57.164498 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.164470    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-dshm" (OuterVolumeSpecName: "dshm") pod "838d15a6-b734-4b5d-9cb5-1676591f0d33" (UID: "838d15a6-b734-4b5d-9cb5-1676591f0d33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:57.164597 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.164532    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/838d15a6-b734-4b5d-9cb5-1676591f0d33-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "838d15a6-b734-4b5d-9cb5-1676591f0d33" (UID: "838d15a6-b734-4b5d-9cb5-1676591f0d33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:31:57.164717 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.164694    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838d15a6-b734-4b5d-9cb5-1676591f0d33-kube-api-access-44wfv" (OuterVolumeSpecName: "kube-api-access-44wfv") pod "838d15a6-b734-4b5d-9cb5-1676591f0d33" (UID: "838d15a6-b734-4b5d-9cb5-1676591f0d33"). InnerVolumeSpecName "kube-api-access-44wfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:31:57.176938 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.176910    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "838d15a6-b734-4b5d-9cb5-1676591f0d33" (UID: "838d15a6-b734-4b5d-9cb5-1676591f0d33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:57.189596 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.189562    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "838d15a6-b734-4b5d-9cb5-1676591f0d33" (UID: "838d15a6-b734-4b5d-9cb5-1676591f0d33"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:57.201216 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.201191    2557 generic.go:356] "Generic (PLEG): container finished" podID="d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b" containerID="c1190795db903db2d35b9313428b7d7e95b57761ad820516f4a2f5d2c507a8a9" exitCode=0
Sep 05 03:31:57.201311 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.201264    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj" event={"ID":"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b","Type":"ContainerDied","Data":"c1190795db903db2d35b9313428b7d7e95b57761ad820516f4a2f5d2c507a8a9"}
Sep 05 03:31:57.203043 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.203021    2557 generic.go:356] "Generic (PLEG): container finished" podID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerID="a7006a05fde8c03d9d6c3df766bfefa28b1ee1a70893f58df94b1e3b59ace7b8" exitCode=0
Sep 05 03:31:57.203119 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.203075    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"
Sep 05 03:31:57.203119 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.203101    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" event={"ID":"838d15a6-b734-4b5d-9cb5-1676591f0d33","Type":"ContainerDied","Data":"a7006a05fde8c03d9d6c3df766bfefa28b1ee1a70893f58df94b1e3b59ace7b8"}
Sep 05 03:31:57.203224 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.203141    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2" event={"ID":"838d15a6-b734-4b5d-9cb5-1676591f0d33","Type":"ContainerDied","Data":"cb1c2d93e1b53711d1c10a230e7382f8b2338ccd53cbdd5c71c35995ed46fb29"}
Sep 05 03:31:57.203224 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.203172    2557 scope.go:117] "RemoveContainer" containerID="a7006a05fde8c03d9d6c3df766bfefa28b1ee1a70893f58df94b1e3b59ace7b8"
Sep 05 03:31:57.212619 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.212600    2557 scope.go:117] "RemoveContainer" containerID="5d2f7abc02b81bbaf5adb83a2876746efc0591f0eff6e9a0ba8f91ae7c50c0f0"
Sep 05 03:31:57.221019 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.220999    2557 scope.go:117] "RemoveContainer" containerID="a7006a05fde8c03d9d6c3df766bfefa28b1ee1a70893f58df94b1e3b59ace7b8"
Sep 05 03:31:57.221289 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:31:57.221269    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7006a05fde8c03d9d6c3df766bfefa28b1ee1a70893f58df94b1e3b59ace7b8\": container with ID starting with a7006a05fde8c03d9d6c3df766bfefa28b1ee1a70893f58df94b1e3b59ace7b8 not found: ID does not exist" containerID="a7006a05fde8c03d9d6c3df766bfefa28b1ee1a70893f58df94b1e3b59ace7b8"
Sep 05 03:31:57.221334 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.221300    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7006a05fde8c03d9d6c3df766bfefa28b1ee1a70893f58df94b1e3b59ace7b8"} err="failed to get container status \"a7006a05fde8c03d9d6c3df766bfefa28b1ee1a70893f58df94b1e3b59ace7b8\": rpc error: code = NotFound desc = could not find container \"a7006a05fde8c03d9d6c3df766bfefa28b1ee1a70893f58df94b1e3b59ace7b8\": container with ID starting with a7006a05fde8c03d9d6c3df766bfefa28b1ee1a70893f58df94b1e3b59ace7b8 not found: ID does not exist"
Sep 05 03:31:57.221334 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.221317    2557 scope.go:117] "RemoveContainer" containerID="5d2f7abc02b81bbaf5adb83a2876746efc0591f0eff6e9a0ba8f91ae7c50c0f0"
Sep 05 03:31:57.221668 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:31:57.221575    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d2f7abc02b81bbaf5adb83a2876746efc0591f0eff6e9a0ba8f91ae7c50c0f0\": container with ID starting with 5d2f7abc02b81bbaf5adb83a2876746efc0591f0eff6e9a0ba8f91ae7c50c0f0 not found: ID does not exist" containerID="5d2f7abc02b81bbaf5adb83a2876746efc0591f0eff6e9a0ba8f91ae7c50c0f0"
Sep 05 03:31:57.221752 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.221677    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2f7abc02b81bbaf5adb83a2876746efc0591f0eff6e9a0ba8f91ae7c50c0f0"} err="failed to get container status \"5d2f7abc02b81bbaf5adb83a2876746efc0591f0eff6e9a0ba8f91ae7c50c0f0\": rpc error: code = NotFound desc = could not find container \"5d2f7abc02b81bbaf5adb83a2876746efc0591f0eff6e9a0ba8f91ae7c50c0f0\": container with ID starting with 5d2f7abc02b81bbaf5adb83a2876746efc0591f0eff6e9a0ba8f91ae7c50c0f0 not found: ID does not exist"
Sep 05 03:31:57.226837 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.226811    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"]
Sep 05 03:31:57.230606 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.230582    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-7787c46b47-mnhl2"]
Sep 05 03:31:57.263203 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.263174    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838d15a6-b734-4b5d-9cb5-1676591f0d33-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:57.263203 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.263200    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44wfv\" (UniqueName: \"kubernetes.io/projected/838d15a6-b734-4b5d-9cb5-1676591f0d33-kube-api-access-44wfv\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:57.263342 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.263211    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:57.263342 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.263223    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:57.263342 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.263231    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-kserve-provision-location\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:57.263342 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.263240    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838d15a6-b734-4b5d-9cb5-1676591f0d33-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:57.672623 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.672593    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:57.767673 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.767635    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg2fv\" (UniqueName: \"kubernetes.io/projected/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-kube-api-access-qg2fv\") pod \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") "
Sep 05 03:31:57.767872 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.767690    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-home\") pod \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") "
Sep 05 03:31:57.767872 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.767718    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-tls-certs\") pod \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") "
Sep 05 03:31:57.767872 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.767740    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-model-cache\") pod \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") "
Sep 05 03:31:57.767872 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.767787    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-dshm\") pod \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") "
Sep 05 03:31:57.767872 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.767863    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-tmp-dir\") pod \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\" (UID: \"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b\") "
Sep 05 03:31:57.768153 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.768048    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-home" (OuterVolumeSpecName: "home") pod "d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b" (UID: "d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:57.768153 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.768056    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b" (UID: "d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:57.768153 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.768134    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-home\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:57.768348 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.768324    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b" (UID: "d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:57.770086 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.770061    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-dshm" (OuterVolumeSpecName: "dshm") pod "d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b" (UID: "d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:31:57.770363 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.770347    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b" (UID: "d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 03:31:57.770363 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.770356    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-kube-api-access-qg2fv" (OuterVolumeSpecName: "kube-api-access-qg2fv") pod "d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b" (UID: "d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b"). InnerVolumeSpecName "kube-api-access-qg2fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:31:57.869531 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.869492    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg2fv\" (UniqueName: \"kubernetes.io/projected/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-kube-api-access-qg2fv\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:57.869531 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.869527    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-tls-certs\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:57.869754 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.869544    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-model-cache\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:57.869754 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.869555    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-dshm\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:57.869754 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:57.869565    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b-tmp-dir\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:31:58.216885 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:58.216794    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"
Sep 05 03:31:58.217053 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:58.216802    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj" event={"ID":"d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b","Type":"ContainerDied","Data":"3dd5332cc6eb9bc7643f64503ab9c1c18ea1f251f98447c2ccc1f053682f31b8"}
Sep 05 03:31:58.217053 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:58.216972    2557 scope.go:117] "RemoveContainer" containerID="c1190795db903db2d35b9313428b7d7e95b57761ad820516f4a2f5d2c507a8a9"
Sep 05 03:31:58.242318 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:58.242280    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"]
Sep 05 03:31:58.244516 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:58.244486    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-bkrsj"]
Sep 05 03:31:58.816923 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:58.816886    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:31:58.915423 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:58.915368    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:31:58.990335 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:58.990270    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" path="/var/lib/kubelet/pods/838d15a6-b734-4b5d-9cb5-1676591f0d33/volumes"
Sep 05 03:31:58.990922 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:31:58.990906    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b" path="/var/lib/kubelet/pods/d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b/volumes"
Sep 05 03:32:47.297182 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:32:47.297130    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:32:53.174416 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:32:53.174374    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:32:58.648197 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:32:58.648155    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:32:58.708130 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:32:58.708082    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:33:02.898098 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:02.898056    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:33:04.469116 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.469075    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nsxxh/must-gather-tsvb8"]
Sep 05 03:33:04.469572 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.469457    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b" containerName="main"
Sep 05 03:33:04.469572 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.469471    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b" containerName="main"
Sep 05 03:33:04.469572 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.469481    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main"
Sep 05 03:33:04.469572 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.469488    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main"
Sep 05 03:33:04.469572 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.469499    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="storage-initializer"
Sep 05 03:33:04.469572 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.469505    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="storage-initializer"
Sep 05 03:33:04.469572 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.469563    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d84efea3-36c1-47a7-8e2f-c8afc0ce6e6b" containerName="main"
Sep 05 03:33:04.469572 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.469576    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="838d15a6-b734-4b5d-9cb5-1676591f0d33" containerName="main"
Sep 05 03:33:04.472771 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.472750    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsxxh/must-gather-tsvb8"
Sep 05 03:33:04.476846 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.476814    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nsxxh\"/\"kube-root-ca.crt\""
Sep 05 03:33:04.477639 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.477617    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nsxxh\"/\"openshift-service-ca.crt\""
Sep 05 03:33:04.477723 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.477622    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nsxxh\"/\"default-dockercfg-btqwt\""
Sep 05 03:33:04.481858 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.481822    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nsxxh/must-gather-tsvb8"]
Sep 05 03:33:04.485280 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.485243    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:33:04.583903 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.583867    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:33:04.647069 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.647025    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f-must-gather-output\") pod \"must-gather-tsvb8\" (UID: \"bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f\") " pod="openshift-must-gather-nsxxh/must-gather-tsvb8"
Sep 05 03:33:04.647273 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.647082    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49mzv\" (UniqueName: \"kubernetes.io/projected/bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f-kube-api-access-49mzv\") pod \"must-gather-tsvb8\" (UID: \"bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f\") " pod="openshift-must-gather-nsxxh/must-gather-tsvb8"
Sep 05 03:33:04.748563 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.748418    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f-must-gather-output\") pod \"must-gather-tsvb8\" (UID: \"bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f\") " pod="openshift-must-gather-nsxxh/must-gather-tsvb8"
Sep 05 03:33:04.748563 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.748508    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49mzv\" (UniqueName: \"kubernetes.io/projected/bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f-kube-api-access-49mzv\") pod \"must-gather-tsvb8\" (UID: \"bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f\") " pod="openshift-must-gather-nsxxh/must-gather-tsvb8"
Sep 05 03:33:04.748802 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.748778    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f-must-gather-output\") pod \"must-gather-tsvb8\" (UID: \"bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f\") " pod="openshift-must-gather-nsxxh/must-gather-tsvb8"
Sep 05 03:33:04.756276 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.756248    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49mzv\" (UniqueName: \"kubernetes.io/projected/bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f-kube-api-access-49mzv\") pod \"must-gather-tsvb8\" (UID: \"bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f\") " pod="openshift-must-gather-nsxxh/must-gather-tsvb8"
Sep 05 03:33:04.783266 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:04.783221    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsxxh/must-gather-tsvb8"
Sep 05 03:33:05.057397 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:05.057363    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nsxxh/must-gather-tsvb8"]
Sep 05 03:33:05.069161 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:05.069127    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 03:33:05.564873 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:05.564828    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsxxh/must-gather-tsvb8" event={"ID":"bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f","Type":"ContainerStarted","Data":"4d50e69b82e6a70ce466a7d3da4cf0ca4d76e7fdfc21290efb9300af6008a6c8"}
Sep 05 03:33:08.954899 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:08.954858    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:33:11.610548 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:11.610503    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsxxh/must-gather-tsvb8" event={"ID":"bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f","Type":"ContainerStarted","Data":"650d50971598e4c10371f51726c806d22b9db8edfbd7de7ec491bce590f41e59"}
Sep 05 03:33:11.611046 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:11.610561    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsxxh/must-gather-tsvb8" event={"ID":"bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f","Type":"ContainerStarted","Data":"415372642fc544815580b181802f2d99d4c89b9f657576aaba9b8b9993226134"}
Sep 05 03:33:11.626815 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:11.626752    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nsxxh/must-gather-tsvb8" podStartSLOduration=1.436389166 podStartE2EDuration="7.626734086s" totalImagesPullingTime="6.19034492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:33:11.625614777 +0000 UTC m=+4495.180771090" watchObservedRunningTime="2026-09-05 03:33:11.626734086 +0000 UTC m=+4495.181890395"
Sep 05 03:33:14.416432 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:14.416384    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:33:14.479678 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:14.479634    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:33:17.422021 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:17.421989    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:33:17.434889 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:17.434861    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:33:27.930953 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:27.930916    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm_82df5585-ce83-449d-8931-c22fd607d1f4/istio-proxy/0.log"
Sep 05 03:33:28.026144 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:28.026099    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm_82df5585-ce83-449d-8931-c22fd607d1f4/istio-proxy/1.log"
Sep 05 03:33:29.173022 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:29.172991    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm_82df5585-ce83-449d-8931-c22fd607d1f4/istio-proxy/0.log"
Sep 05 03:33:29.269829 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:29.269781    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm_82df5585-ce83-449d-8931-c22fd607d1f4/istio-proxy/1.log"
Sep 05 03:33:30.785378 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:30.785337    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-9cmdj_3159908c-3142-42b9-b97d-590ea1610462/manager/0.log"
Sep 05 03:33:30.794995 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:30.794964    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-6sxln_31b61bd5-b792-4925-a4ff-94ccbf297f78/limitador/0.log"
Sep 05 03:33:30.815199 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:30.815166    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-czc6z_679a80f2-3028-4d51-a687-e2d5d7a36341/manager/0.log"
Sep 05 03:33:34.752321 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:34.752276    2557 generic.go:356] "Generic (PLEG): container finished" podID="bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" containerID="415372642fc544815580b181802f2d99d4c89b9f657576aaba9b8b9993226134" exitCode=0
Sep 05 03:33:34.752764 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:34.752349    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsxxh/must-gather-tsvb8" event={"ID":"bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f","Type":"ContainerDied","Data":"415372642fc544815580b181802f2d99d4c89b9f657576aaba9b8b9993226134"}
Sep 05 03:33:34.752812 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:34.752759    2557 scope.go:117] "RemoveContainer" containerID="415372642fc544815580b181802f2d99d4c89b9f657576aaba9b8b9993226134"
Sep 05 03:33:35.635305 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:35.635271    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-nsxxh_must-gather-tsvb8_bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f/gather/0.log"
Sep 05 03:33:36.158842 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.158801    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:33:36.317124 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.317079    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g75mm/must-gather-bndh5"]
Sep 05 03:33:36.320287 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.320261    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g75mm/must-gather-bndh5"
Sep 05 03:33:36.322370 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.322342    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g75mm\"/\"kube-root-ca.crt\""
Sep 05 03:33:36.322957 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.322941    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g75mm\"/\"openshift-service-ca.crt\""
Sep 05 03:33:36.323354 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.323337    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g75mm\"/\"default-dockercfg-js4pb\""
Sep 05 03:33:36.329971 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.329947    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g75mm/must-gather-bndh5"]
Sep 05 03:33:36.340011 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.339980    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:33:36.410929 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.410831    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:33:36.432666 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.432638    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlt8p\" (UniqueName: \"kubernetes.io/projected/7f28fce6-4ef9-40b6-9db0-31bc80401e94-kube-api-access-jlt8p\") pod \"must-gather-bndh5\" (UID: \"7f28fce6-4ef9-40b6-9db0-31bc80401e94\") " pod="openshift-must-gather-g75mm/must-gather-bndh5"
Sep 05 03:33:36.432771 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.432698    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7f28fce6-4ef9-40b6-9db0-31bc80401e94-must-gather-output\") pod \"must-gather-bndh5\" (UID: \"7f28fce6-4ef9-40b6-9db0-31bc80401e94\") " pod="openshift-must-gather-g75mm/must-gather-bndh5"
Sep 05 03:33:36.533657 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.533616    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7f28fce6-4ef9-40b6-9db0-31bc80401e94-must-gather-output\") pod \"must-gather-bndh5\" (UID: \"7f28fce6-4ef9-40b6-9db0-31bc80401e94\") " pod="openshift-must-gather-g75mm/must-gather-bndh5"
Sep 05 03:33:36.533859 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.533703    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlt8p\" (UniqueName: \"kubernetes.io/projected/7f28fce6-4ef9-40b6-9db0-31bc80401e94-kube-api-access-jlt8p\") pod \"must-gather-bndh5\" (UID: \"7f28fce6-4ef9-40b6-9db0-31bc80401e94\") " pod="openshift-must-gather-g75mm/must-gather-bndh5"
Sep 05 03:33:36.534046 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.534023    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7f28fce6-4ef9-40b6-9db0-31bc80401e94-must-gather-output\") pod \"must-gather-bndh5\" (UID: \"7f28fce6-4ef9-40b6-9db0-31bc80401e94\") " pod="openshift-must-gather-g75mm/must-gather-bndh5"
Sep 05 03:33:36.542024 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.541997    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlt8p\" (UniqueName: \"kubernetes.io/projected/7f28fce6-4ef9-40b6-9db0-31bc80401e94-kube-api-access-jlt8p\") pod \"must-gather-bndh5\" (UID: \"7f28fce6-4ef9-40b6-9db0-31bc80401e94\") " pod="openshift-must-gather-g75mm/must-gather-bndh5"
Sep 05 03:33:36.630190 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.630150    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g75mm/must-gather-bndh5"
Sep 05 03:33:36.900101 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:36.900071    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g75mm/must-gather-bndh5"]
Sep 05 03:33:36.902111 ip-10-0-143-128 kubenswrapper[2557]: W0905 03:33:36.902082    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f28fce6_4ef9_40b6_9db0_31bc80401e94.slice/crio-5220aa271a9b221784e5acd4b84212832ab8683c2913e48c2c800554296c1218 WatchSource:0}: Error finding container 5220aa271a9b221784e5acd4b84212832ab8683c2913e48c2c800554296c1218: Status 404 returned error can't find the container with id 5220aa271a9b221784e5acd4b84212832ab8683c2913e48c2c800554296c1218
Sep 05 03:33:37.773493 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:37.773420    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g75mm/must-gather-bndh5" event={"ID":"7f28fce6-4ef9-40b6-9db0-31bc80401e94","Type":"ContainerStarted","Data":"ea96ceb042708f5da97e52dc760f8ccd23f9fa51374aa9d5c8c94c556c185569"}
Sep 05 03:33:37.773952 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:37.773508    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g75mm/must-gather-bndh5" event={"ID":"7f28fce6-4ef9-40b6-9db0-31bc80401e94","Type":"ContainerStarted","Data":"5220aa271a9b221784e5acd4b84212832ab8683c2913e48c2c800554296c1218"}
Sep 05 03:33:38.784464 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:38.784399    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g75mm/must-gather-bndh5" event={"ID":"7f28fce6-4ef9-40b6-9db0-31bc80401e94","Type":"ContainerStarted","Data":"d63931d5615e05ee8ad8cbf3ae9605996b4e0bed2a7f7a7b1717177756087428"}
Sep 05 03:33:38.804405 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:38.804330    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g75mm/must-gather-bndh5" podStartSLOduration=2.094739012 podStartE2EDuration="2.80430825s" totalImagesPullingTime="709.569238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 03:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:33:38.80062363 +0000 UTC m=+4522.355779941" watchObservedRunningTime="2026-09-05 03:33:38.80430825 +0000 UTC m=+4522.359464560"
Sep 05 03:33:39.339212 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:39.339168    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cqpw2_39f296d4-d2f6-46c7-ad0d-b520ecb0138e/global-pull-secret-syncer/0.log"
Sep 05 03:33:39.427511 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:39.427469    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log"
Sep 05 03:33:39.565677 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:39.565637    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-128.ec2.internal_ea53f2068cb53a80ffb0922aa7c95e21/haproxy/0.log"
Sep 05 03:33:41.144746 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.144690    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-nsxxh/must-gather-tsvb8"]
Sep 05 03:33:41.145297 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.144941    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-nsxxh/must-gather-tsvb8" podUID="bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" containerName="copy" containerID="cri-o://650d50971598e4c10371f51726c806d22b9db8edfbd7de7ec491bce590f41e59" gracePeriod=2
Sep 05 03:33:41.151228 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:33:41.151182    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tsvb8\" is forbidden: User \"system:node:ip-10-0-143-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-nsxxh\": no relationship found between node 'ip-10-0-143-128.ec2.internal' and this object" podUID="bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" pod="openshift-must-gather-nsxxh/must-gather-tsvb8"
Sep 05 03:33:41.152987 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.152949    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-nsxxh/must-gather-tsvb8"]
Sep 05 03:33:41.569457 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.568703    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-nsxxh_must-gather-tsvb8_bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f/copy/0.log"
Sep 05 03:33:41.574697 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.573783    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsxxh/must-gather-tsvb8"
Sep 05 03:33:41.577704 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:33:41.577664    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tsvb8\" is forbidden: User \"system:node:ip-10-0-143-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-nsxxh\": no relationship found between node 'ip-10-0-143-128.ec2.internal' and this object" podUID="bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" pod="openshift-must-gather-nsxxh/must-gather-tsvb8"
Sep 05 03:33:41.692722 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.692629    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f-must-gather-output\") pod \"bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f\" (UID: \"bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f\") "
Sep 05 03:33:41.703138 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.703097    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49mzv\" (UniqueName: \"kubernetes.io/projected/bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f-kube-api-access-49mzv\") pod \"bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f\" (UID: \"bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f\") "
Sep 05 03:33:41.704640 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.704604    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" (UID: "bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 03:33:41.707950 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.707904    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f-kube-api-access-49mzv" (OuterVolumeSpecName: "kube-api-access-49mzv") pod "bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" (UID: "bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f"). InnerVolumeSpecName "kube-api-access-49mzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 03:33:41.805989 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.805927    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-nsxxh_must-gather-tsvb8_bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f/copy/0.log"
Sep 05 03:33:41.806767 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.806714    2557 generic.go:356] "Generic (PLEG): container finished" podID="bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" containerID="650d50971598e4c10371f51726c806d22b9db8edfbd7de7ec491bce590f41e59" exitCode=143
Sep 05 03:33:41.807012 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.806975    2557 scope.go:117] "RemoveContainer" containerID="650d50971598e4c10371f51726c806d22b9db8edfbd7de7ec491bce590f41e59"
Sep 05 03:33:41.807421 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.807323    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsxxh/must-gather-tsvb8"
Sep 05 03:33:41.817664 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.817606    2557 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f-must-gather-output\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:33:41.817664 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.817639    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49mzv\" (UniqueName: \"kubernetes.io/projected/bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f-kube-api-access-49mzv\") on node \"ip-10-0-143-128.ec2.internal\" DevicePath \"\""
Sep 05 03:33:41.819528 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:33:41.819483    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tsvb8\" is forbidden: User \"system:node:ip-10-0-143-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-nsxxh\": no relationship found between node 'ip-10-0-143-128.ec2.internal' and this object" podUID="bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" pod="openshift-must-gather-nsxxh/must-gather-tsvb8"
Sep 05 03:33:41.831669 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:33:41.831626    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tsvb8\" is forbidden: User \"system:node:ip-10-0-143-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-nsxxh\": no relationship found between node 'ip-10-0-143-128.ec2.internal' and this object" podUID="bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" pod="openshift-must-gather-nsxxh/must-gather-tsvb8"
Sep 05 03:33:41.832083 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.832042    2557 scope.go:117] "RemoveContainer" containerID="415372642fc544815580b181802f2d99d4c89b9f657576aaba9b8b9993226134"
Sep 05 03:33:41.853606 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.853014    2557 scope.go:117] "RemoveContainer" containerID="650d50971598e4c10371f51726c806d22b9db8edfbd7de7ec491bce590f41e59"
Sep 05 03:33:41.853606 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:33:41.853476    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650d50971598e4c10371f51726c806d22b9db8edfbd7de7ec491bce590f41e59\": container with ID starting with 650d50971598e4c10371f51726c806d22b9db8edfbd7de7ec491bce590f41e59 not found: ID does not exist" containerID="650d50971598e4c10371f51726c806d22b9db8edfbd7de7ec491bce590f41e59"
Sep 05 03:33:41.853606 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.853510    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650d50971598e4c10371f51726c806d22b9db8edfbd7de7ec491bce590f41e59"} err="failed to get container status \"650d50971598e4c10371f51726c806d22b9db8edfbd7de7ec491bce590f41e59\": rpc error: code = NotFound desc = could not find container \"650d50971598e4c10371f51726c806d22b9db8edfbd7de7ec491bce590f41e59\": container with ID starting with 650d50971598e4c10371f51726c806d22b9db8edfbd7de7ec491bce590f41e59 not found: ID does not exist"
Sep 05 03:33:41.853606 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.853537    2557 scope.go:117] "RemoveContainer" containerID="415372642fc544815580b181802f2d99d4c89b9f657576aaba9b8b9993226134"
Sep 05 03:33:41.854214 ip-10-0-143-128 kubenswrapper[2557]: E0905 03:33:41.854133    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415372642fc544815580b181802f2d99d4c89b9f657576aaba9b8b9993226134\": container with ID starting with 415372642fc544815580b181802f2d99d4c89b9f657576aaba9b8b9993226134 not found: ID does not exist" containerID="415372642fc544815580b181802f2d99d4c89b9f657576aaba9b8b9993226134"
Sep 05 03:33:41.854214 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:41.854164    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415372642fc544815580b181802f2d99d4c89b9f657576aaba9b8b9993226134"} err="failed to get container status \"415372642fc544815580b181802f2d99d4c89b9f657576aaba9b8b9993226134\": rpc error: code = NotFound desc = could not find container \"415372642fc544815580b181802f2d99d4c89b9f657576aaba9b8b9993226134\": container with ID starting with 415372642fc544815580b181802f2d99d4c89b9f657576aaba9b8b9993226134 not found: ID does not exist"
Sep 05 03:33:42.023034 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:42.022784    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:33:42.122425 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:42.122372    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:33:42.995718 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:42.995675    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" path="/var/lib/kubelet/pods/bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f/volumes"
Sep 05 03:33:44.887463 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:44.885049    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-9cmdj_3159908c-3142-42b9-b97d-590ea1610462/manager/0.log"
Sep 05 03:33:44.918309 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:44.918273    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-6sxln_31b61bd5-b792-4925-a4ff-94ccbf297f78/limitador/0.log"
Sep 05 03:33:44.960241 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:44.960198    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-czc6z_679a80f2-3028-4d51-a687-e2d5d7a36341/manager/0.log"
Sep 05 03:33:46.328141 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:46.328101    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:33:46.387297 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:46.387258    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dvtdd_43bb8d7d-c160-4ad9-85d5-6b07e88dfd31/konnectivity-agent/0.log" limit=1024
Sep 05 03:33:46.985663 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:46.985622    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6dmz_d010c7b3-2a2d-4aef-b279-2948b7f0b0f9/node-exporter/0.log"
Sep 05 03:33:47.031468 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:47.030045    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6dmz_d010c7b3-2a2d-4aef-b279-2948b7f0b0f9/kube-rbac-proxy/0.log"
Sep 05 03:33:47.061470 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:47.061416    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6dmz_d010c7b3-2a2d-4aef-b279-2948b7f0b0f9/init-textfile/0.log"
Sep 05 03:33:47.513747 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:47.513693    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78667cd95d-g9snm_11424145-7ce8-41b5-8bdb-44046dfdded2/telemeter-client/0.log"
Sep 05 03:33:47.540107 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:47.540077    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78667cd95d-g9snm_11424145-7ce8-41b5-8bdb-44046dfdded2/reload/0.log"
Sep 05 03:33:47.570008 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:47.569969    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78667cd95d-g9snm_11424145-7ce8-41b5-8bdb-44046dfdded2/kube-rbac-proxy/0.log"
Sep 05 03:33:49.478360 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:49.478328    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/2.log"
Sep 05 03:33:49.487347 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:49.487307    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-gtf6d_83157c8a-294b-4add-aa27-3db843d0a9e5/console-operator/3.log"
Sep 05 03:33:49.981683 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:49.981650    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-868fdc5cdd-jtvkn_db98bf51-1b00-40cc-ba86-3f7aef2f8a5b/console/0.log"
Sep 05 03:33:50.015860 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.015815    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"]
Sep 05 03:33:50.016248 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.016231    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" containerName="copy"
Sep 05 03:33:50.016332 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.016250    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" containerName="copy"
Sep 05 03:33:50.016332 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.016276    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" containerName="gather"
Sep 05 03:33:50.016332 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.016282    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" containerName="gather"
Sep 05 03:33:50.016532 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.016338    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" containerName="copy"
Sep 05 03:33:50.016532 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.016349    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac92ce3-22b9-4a6f-80de-10fb3d2f5f7f" containerName="gather"
Sep 05 03:33:50.019371 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.019345    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.029987 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.029933    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"]
Sep 05 03:33:50.093999 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.093959    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-lib-modules\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.094203 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.094016    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rplt\" (UniqueName: \"kubernetes.io/projected/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-kube-api-access-7rplt\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.094203 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.094094    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-sys\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.094203 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.094140    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-podres\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.094203 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.094196    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-proc\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.195052 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.195005    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-podres\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.195052 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.195059    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-proc\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.195355 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.195104    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-lib-modules\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.195355 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.195136    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rplt\" (UniqueName: \"kubernetes.io/projected/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-kube-api-access-7rplt\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.195355 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.195179    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-sys\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.195355 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.195224    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-proc\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.195355 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.195225    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-podres\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.195355 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.195282    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-lib-modules\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.195355 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.195283    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-sys\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.203206 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.203158    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rplt\" (UniqueName: \"kubernetes.io/projected/d625ddf3-4ce4-4d6b-a12f-36b3a37a538c-kube-api-access-7rplt\") pod \"perf-node-gather-daemonset-zfgps\" (UID: \"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c\") " pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.336023 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.335979    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.622533 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.622407    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"]
Sep 05 03:33:50.899766 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.899665    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps" event={"ID":"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c","Type":"ContainerStarted","Data":"6ee8f60c1b290718328ea2bda732cae6267cf5419ce9b994bac4601dc345d4f1"}
Sep 05 03:33:50.899766 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.899733    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps" event={"ID":"d625ddf3-4ce4-4d6b-a12f-36b3a37a538c","Type":"ContainerStarted","Data":"72330963b859c7a9f1f4d78334f261749d46a19be6c3439c429b111e7b2fde0d"}
Sep 05 03:33:50.899999 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.899884    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:50.915814 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:50.915758    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps" podStartSLOduration=1.9157397889999999 podStartE2EDuration="1.915739789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 03:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 03:33:50.914968966 +0000 UTC m=+4534.470125290" watchObservedRunningTime="2026-09-05 03:33:50.915739789 +0000 UTC m=+4534.470896098"
Sep 05 03:33:51.783158 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:51.783097    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qs48v_19ece4dc-ca43-42a0-bf9c-64ab7e233025/dns/0.log"
Sep 05 03:33:51.804171 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:51.804135    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qs48v_19ece4dc-ca43-42a0-bf9c-64ab7e233025/kube-rbac-proxy/0.log"
Sep 05 03:33:51.880826 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:51.880785    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g8xt5_c10e20bb-ff5e-4f8f-82c1-7f12be57bd1b/dns-node-resolver/0.log"
Sep 05 03:33:52.370876 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:52.370841    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56897c4b7f-7m4jc_c3aa1a59-6ce1-4ae7-93d7-d680837d353d/registry/0.log"
Sep 05 03:33:52.406353 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:52.406310    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56897c4b7f-7m4jc_c3aa1a59-6ce1-4ae7-93d7-d680837d353d/registry/1.log"
Sep 05 03:33:52.460235 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:52.460196    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lpd6d_2836c4ea-8e45-4bd3-9c96-ace613f40ed2/node-ca/0.log"
Sep 05 03:33:53.814852 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:53.814816    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm_82df5585-ce83-449d-8931-c22fd607d1f4/istio-proxy/0.log"
Sep 05 03:33:53.913083 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:53.913040    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm_82df5585-ce83-449d-8931-c22fd607d1f4/istio-proxy/1.log"
Sep 05 03:33:54.450299 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:54.450255    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5t47h_c7310203-e7ca-467e-945c-d4a9db60e8fd/serve-healthcheck-canary/0.log"
Sep 05 03:33:54.964394 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:54.964343    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-tnh5c_d16b566d-44e4-46fe-bf78-6bb6df91a954/insights-operator/0.log"
Sep 05 03:33:54.967793 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:54.967724    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-tnh5c_d16b566d-44e4-46fe-bf78-6bb6df91a954/insights-operator/1.log"
Sep 05 03:33:55.122127 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:55.122089    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sdzmq_b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b/kube-rbac-proxy/0.log"
Sep 05 03:33:55.144111 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:55.144068    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sdzmq_b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b/exporter/0.log"
Sep 05 03:33:55.166397 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:55.166358    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sdzmq_b0ffd5ff-67dd-4937-bdbe-5b5f1e45ca1b/extractor/0.log"
Sep 05 03:33:56.919633 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:56.919604    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g75mm/perf-node-gather-daemonset-zfgps"
Sep 05 03:33:58.790256 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:58.790221    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-577b5fb6f8-tjpj4_44a6af99-2320-483c-af78-37c506467fc4/manager/0.log"
Sep 05 03:33:59.423965 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:59.423920    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-7bcc44dcb6-nndsk_0bfb153b-7f0e-4e59-80d8-4f7bd5bbae82/manager/0.log"
Sep 05 03:33:59.445757 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:33:59.445721    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-l874f_ed5fbaa7-2a65-49e8-91c6-23ad1d73e63b/server/0.log"
Sep 05 03:34:01.342224 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:01.342183    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-cd6lx_5233c7a9-331e-49ef-9cc9-5c99037d4c49/manager/0.log"
Sep 05 03:34:01.426864 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:01.426826    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pggpg_dcc86525-561c-4d9f-87f9-0df00516dc33/s3-init/0.log"
Sep 05 03:34:01.444160 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:01.444126    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pggpg_dcc86525-561c-4d9f-87f9-0df00516dc33/download-hf-model/0.log"
Sep 05 03:34:01.474239 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:01.474207    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-zdgtr_bc5ad237-ad5c-4817-ba9a-148cfd194e99/seaweedfs/0.log"
Sep 05 03:34:01.751342 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:01.751300    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm_82df5585-ce83-449d-8931-c22fd607d1f4/istio-proxy/0.log"
Sep 05 03:34:01.849559 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:01.849517    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm_82df5585-ce83-449d-8931-c22fd607d1f4/istio-proxy/1.log"
Sep 05 03:34:03.333532 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:03.333487    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm_82df5585-ce83-449d-8931-c22fd607d1f4/istio-proxy/0.log"
Sep 05 03:34:03.429459 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:03.429406    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-7p2mm_82df5585-ce83-449d-8931-c22fd607d1f4/istio-proxy/1.log"
Sep 05 03:34:06.639175 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:06.639130    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-54qc4_12667c29-3724-4078-964d-07268713477a/kube-storage-version-migrator-operator/1.log"
Sep 05 03:34:06.702699 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:06.702654    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-54qc4_12667c29-3724-4078-964d-07268713477a/kube-storage-version-migrator-operator/0.log"
Sep 05 03:34:07.610886 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:07.610856    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j9fmm_a78d01f0-d41b-40ad-8d73-ddc3df30e814/kube-multus-additional-cni-plugins/0.log"
Sep 05 03:34:07.631932 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:07.631898    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j9fmm_a78d01f0-d41b-40ad-8d73-ddc3df30e814/egress-router-binary-copy/0.log"
Sep 05 03:34:07.660157 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:07.660123    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j9fmm_a78d01f0-d41b-40ad-8d73-ddc3df30e814/cni-plugins/0.log"
Sep 05 03:34:07.679414 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:07.679380    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j9fmm_a78d01f0-d41b-40ad-8d73-ddc3df30e814/bond-cni-plugin/0.log"
Sep 05 03:34:07.699783 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:07.699743    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j9fmm_a78d01f0-d41b-40ad-8d73-ddc3df30e814/routeoverride-cni/0.log"
Sep 05 03:34:07.724044 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:07.724003    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j9fmm_a78d01f0-d41b-40ad-8d73-ddc3df30e814/whereabouts-cni-bincopy/0.log"
Sep 05 03:34:07.743279 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:07.743247    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j9fmm_a78d01f0-d41b-40ad-8d73-ddc3df30e814/whereabouts-cni/0.log"
Sep 05 03:34:08.169362 ip-10-0-143-128 kubenswrapper[2557]: I0905 03:34:08.169321    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gq27j_ad420093-4c18-44bf-b72b-ffa0c853997c/kube-multus/0.log"