Sep 16 15:50:03.372851 ip-10-0-143-143 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 15:50:03.792853 ip-10-0-143-143 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 15:50:03.792853 ip-10-0-143-143 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 15:50:03.792853 ip-10-0-143-143 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 15:50:03.792853 ip-10-0-143-143 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 15:50:03.792853 ip-10-0-143-143 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 15:50:03.794885 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.794790    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 15:50:03.799063 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799045    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 15:50:03.799063 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799063    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799068    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799071    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799074    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799077    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799080    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799083    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799086    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799089    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799091    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799094    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799099    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799103    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799106    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799115    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799118    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799121    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799123    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799126    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799128    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 15:50:03.799133 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799131    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799134    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799136    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799139    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799142    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799145    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799148    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799150    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799153    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799156    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799158    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799161    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799163    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799165    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799168    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799171    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799173    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799176    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799179    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799181    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 15:50:03.799624 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799184    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799186    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799189    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799191    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799194    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799197    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799199    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799203    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799206    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799209    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799213    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799216    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799220    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799223    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799226    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799229    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799232    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799234    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799237    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 15:50:03.800152 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799239    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799242    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799246    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799249    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799252    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799255    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799258    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799260    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799263    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799265    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799268    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799277    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799280    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799283    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799285    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799288    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799292    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799294    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799297    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799299    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 15:50:03.800649 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799302    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799304    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799307    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799309    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799312    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799316    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799319    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799322    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799325    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799327    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799330    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799333    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799336    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799338    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799341    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799343    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799346    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799348    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.799351    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 15:50:03.801138 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800075    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800082    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800085    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800087    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800090    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800093    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800097    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800099    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800102    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800105    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800108    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800110    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800113    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800115    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800118    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800121    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800123    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800126    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800129    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800132    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 15:50:03.801629 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800135    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800138    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800141    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800143    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800147    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800151    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800154    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800157    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800160    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800163    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800165    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800168    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800171    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800174    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800177    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800180    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800182    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800185    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800188    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 15:50:03.802203 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800191    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800194    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800196    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800199    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800202    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800204    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800206    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800209    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800211    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800214    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800216    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800219    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800222    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800226    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800228    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800231    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800233    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800238    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800241    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800244    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 15:50:03.802712 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800246    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800249    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800251    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800254    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800257    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800260    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800262    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800264    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800267    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800269    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800272    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800274    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800277    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800280    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800283    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800285    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800288    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800290    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800293    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800295    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 15:50:03.803214 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800298    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800302    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800305    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800308    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800313    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800317    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800320    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800323    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800326    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800329    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800331    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800335    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800339    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800342    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800345    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800349    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800352    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800356    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800359    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 15:50:03.803719 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800362    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800899    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800913    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800917    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800920    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800923    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800926    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800931    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800935    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800939    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800944    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800948    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800952    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800957    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800959    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800962    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800964    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800967    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800970    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 15:50:03.804200 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800972    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800975    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800978    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800980    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800983    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800985    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800988    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800991    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800994    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800997    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.800999    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801002    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801004    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801008    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801012    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801015    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801019    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801023    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801027    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 15:50:03.804682 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801031    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801036    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801039    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801041    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801044    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801047    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801049    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801052    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801054    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801057    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801060    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801063    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801066    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801069    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801072    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801074    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801077    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801080    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801082    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801085    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801087    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 15:50:03.805159 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801090    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801092    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801095    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801098    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801102    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801106    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801111    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801119    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801122    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801125    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801128    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801130    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801133    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801136    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801138    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801141    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801144    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801146    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801150    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801153    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 15:50:03.805746 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801155    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801159    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801162    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801165    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801167    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801171    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801174    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801176    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801180    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801187    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801193    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801198    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801200    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801203    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801206    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801208    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801211    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801213    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801216    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 15:50:03.806237 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801220    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.801224    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801310    2574 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801348    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801359    2574 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801366    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801371    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801374    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801382    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801386    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801389    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801393    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801396    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801400    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801403    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801406    2574 flags.go:64] FLAG: --cgroup-root=""
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801409    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801412    2574 flags.go:64] FLAG: --client-ca-file=""
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801416    2574 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801419    2574 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801423    2574 flags.go:64] FLAG: --cluster-domain=""
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801427    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801432    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801438    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 15:50:03.806760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801443    2574 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801448    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801451    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801454    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801458    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801461    2574 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801464    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801467    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801472    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801475    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801480    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801484    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801486    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801489    2574 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801493    2574 flags.go:64] FLAG: --enable-server="true"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801495    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801500    2574 flags.go:64] FLAG: --event-burst="100"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801503    2574 flags.go:64] FLAG: --event-qps="50"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801507    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801512    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801530    2574 flags.go:64] FLAG: --eviction-hard=""
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801536    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801540    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801543    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801546    2574 flags.go:64] FLAG: --eviction-soft=""
Sep 16 15:50:03.807348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801553    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801556    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801560    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801562    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801565    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801568    2574 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801571    2574 flags.go:64] FLAG: --feature-gates=""
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801575    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801578    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801581    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801584    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801588    2574 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801591    2574 flags.go:64] FLAG: --help="false"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801595    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-143-143.ec2.internal"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801600    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801607    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801612    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801616    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801620    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801623    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801626    2574 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801629    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801632    2574 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 15:50:03.808001 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801635    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801639    2574 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801642    2574 flags.go:64] FLAG: --kube-reserved=""
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801645    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801648    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801651    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801654    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801657    2574 flags.go:64] FLAG: --lock-file=""
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801660    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801664    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801667    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801673    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801676    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801681    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801685    2574 flags.go:64] FLAG: --logging-format="text"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801690    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801695    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801698    2574 flags.go:64] FLAG: --manifest-url=""
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801701    2574 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801706    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801709    2574 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801713    2574 flags.go:64] FLAG: --max-pods="110"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801716    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801719    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801722    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 15:50:03.808611 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801725    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801728    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801732    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801735    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801743    2574 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801746    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801749    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801752    2574 flags.go:64] FLAG: --pod-cidr=""
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801755    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801763    2574 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801768    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801773    2574 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801776    2574 flags.go:64] FLAG: --port="10250"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801779    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801782    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f7c117b739b9ccbf"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801785    2574 flags.go:64] FLAG: --qos-reserved=""
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801791    2574 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801793    2574 flags.go:64] FLAG: --register-node="true"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801796    2574 flags.go:64] FLAG: --register-with-taints=""
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801800    2574 flags.go:64] FLAG: --registry-burst="10"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801803    2574 flags.go:64] FLAG: --registry-qps="5"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801806    2574 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801809    2574 flags.go:64] FLAG: --reserved-memory=""
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801814    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801817    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 15:50:03.809239 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801820    2574 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801823    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801826    2574 flags.go:64] FLAG: --runonce="false"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801829    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801831    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801834    2574 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801838    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801843    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801848    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801853    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801857    2574 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801861    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801864    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801867    2574 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801869    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801873    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801876    2574 flags.go:64] FLAG: --system-cgroups=""
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801879    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801885    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801887    2574 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801890    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801894    2574 flags.go:64] FLAG: --tls-min-version=""
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801897    2574 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801901    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801904    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 15:50:03.809917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801907    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801910    2574 flags.go:64] FLAG: --v="2"
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801914    2574 flags.go:64] FLAG: --version="false"
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801920    2574 flags.go:64] FLAG: --vmodule=""
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801926    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.801932    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802075    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802082    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802087    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802092    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802096    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802099    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802102    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802105    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802108    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802110    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802113    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802116    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802119    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802122    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802125    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 15:50:03.810600 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802128    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802132    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802136    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802139    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802142    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802144    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802147    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802150    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802152    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802156    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802159    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802164    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802168    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802172    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802176    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802179    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802182    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802185    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802187    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802190    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 15:50:03.811139 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802192    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802195    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802198    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802200    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802203    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802206    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802209    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802212    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802214    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802217    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802219    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802223    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802226    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802228    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802233    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802236    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802240    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802245    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802249    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 15:50:03.811734 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802253    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802257    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802261    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802264    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802267    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802270    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802272    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802275    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802277    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802281    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802284    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802287    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802290    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802293    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802296    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802299    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802304    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802307    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802310    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802312    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 15:50:03.812204 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802315    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802318    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802321    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802325    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802330    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802335    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802339    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802342    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802344    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802347    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802350    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802352    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802355    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802357    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802361    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802364    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802367    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802370    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802372    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802375    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 15:50:03.812735 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802377    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802380    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802382    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802385    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.802387    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.803089    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.811150    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.811171    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.812120    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.812128    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812186    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812192    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812195    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812198    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812202    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 15:50:03.813230 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812205    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812208    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812211    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812214    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812217    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812222    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812227    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812230    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812233    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812236    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812239    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812242    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812245    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812247    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812250    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812252    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812255    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812257    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812260    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 15:50:03.813628 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812263    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812265    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812268    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812271    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812273    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812277    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812280    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812283    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812285    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812288    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812290    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812293    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812296    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812298    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812301    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812304    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812306    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812309    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812311    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812314    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 15:50:03.814137 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812317    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812319    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812322    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812325    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812328    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812330    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812333    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812335    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812338    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812341    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812343    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812346    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812348    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812351    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812355    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812359    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812361    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812364    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812368    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 15:50:03.814637 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812371    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812373    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812376    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812378    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812381    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812384    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812387    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812389    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812392    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812395    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812397    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812400    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812402    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812405    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812407    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812410    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812413    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812415    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812418    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812420    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 15:50:03.815121 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812423    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812425    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812428    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812430    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812433    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812435    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812438    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812442    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812445    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812448    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812451    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812454    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812458    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812461    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812463    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 15:50:03.815623 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812466    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.812471    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812597    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812602    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812605    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812608    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812611    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812614    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812616    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812619    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812622    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812625    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812628    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812631    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812633    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 15:50:03.815994 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812636    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812639    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812641    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812644    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812646    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812649    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812651    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812654    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812656    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812659    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812662    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812664    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812667    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812670    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812673    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812676    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812679    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812681    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812684    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812687    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 15:50:03.816371 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812689    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812692    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812695    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812697    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812700    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812702    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812705    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812708    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812710    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812714    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812719    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812723    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812726    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812729    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812732    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812734    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812737    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812740    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812743    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 15:50:03.816892 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812746    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812749    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812752    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812755    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812758    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812761    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812764    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812767    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812770    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812773    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812775    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812778    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812780    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812783    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812786    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812788    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812791    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812793    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812796    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812798    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 15:50:03.817373 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812800    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812803    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812806    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812809    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812811    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812814    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812816    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812819    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812821    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812824    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812826    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812829    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812832    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812834    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812836    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812839    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812842    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812844    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812851    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 15:50:03.817879 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812854    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 15:50:03.818343 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812858    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 15:50:03.818343 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812861    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 15:50:03.818343 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812864    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 15:50:03.818343 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812867    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 15:50:03.818343 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812870    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 15:50:03.818343 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812872    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 15:50:03.818343 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:03.812875    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 15:50:03.818343 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.812880    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 15:50:03.818343 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.813002    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 15:50:03.818343 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.815495    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 15:50:03.818343 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.816662    2574 server.go:1019] "Starting client certificate rotation"
Sep 16 15:50:03.818343 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.816764    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 15:50:03.818343 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.816836    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 15:50:03.821536 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.821465    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 15:50:03.827651 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.827632    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 15:50:03.835601 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.835578    2574 log.go:25] "Validated CRI v1 runtime API"
Sep 16 15:50:03.839144 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.839132    2574 log.go:25] "Validated CRI v1 image API"
Sep 16 15:50:03.840388 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.840364    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 15:50:03.844240 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.844219    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 15:50:03.846733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.846709    2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c404c69c-4c37-4fba-8bbb-703ee3d4362d:/dev/nvme0n1p3 eb0c072e-ab88-473f-8b38-e21aeb3384bf:/dev/nvme0n1p4]
Sep 16 15:50:03.846824 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.846731    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 15:50:03.853070 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.852934    2574 manager.go:216] Machine: {Timestamp:2026-09-16 15:50:03.851485536 +0000 UTC m=+0.371108304 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102339 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cef3ec35ccf520dbcd68e040ebfa6 SystemUUID:ec2cef3e-c35c-cf52-0dbc-d68e040ebfa6 BootID:2061b4c6-2606-4b4a-bc40-9db5fecf34a9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f7:41:5e:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f7:41:5e:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:6b:72:33:eb:34 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 15:50:03.853070 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.853056    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 15:50:03.853240 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.853171    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 15:50:03.856887 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.856862    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 15:50:03.857026 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.856890    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-143.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 15:50:03.858491 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.858481    2574 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 15:50:03.858538 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.858493    2574 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 15:50:03.858538 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.858501    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 15:50:03.859421 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.859409    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 15:50:03.860716 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.860706    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 15:50:03.865461 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.865449    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 15:50:03.865592 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.865583    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 15:50:03.865640 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.865622    2574 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 15:50:03.865640 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.865636    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 15:50:03.865712 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.865652    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 15:50:03.865712 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.865661    2574 kubelet.go:398] "Adding apiserver pod source"
Sep 16 15:50:03.865712 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.865669    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 15:50:03.866136 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.866119    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 15:50:03.866189 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.866153    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 15:50:03.866306 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.866286    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z98sx"
Sep 16 15:50:03.870286 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.870263    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 15:50:03.873267 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.873253    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 15:50:03.873316 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.873284    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 15:50:03.874017 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.874001    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z98sx"
Sep 16 15:50:03.875427 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:03.875392    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 15:50:03.875427 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:03.875401    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 15:50:03.876837 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.876819    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 15:50:03.876909 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.876846    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 15:50:03.876909 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.876854    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 15:50:03.876909 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.876859    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 15:50:03.876909 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.876868    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 15:50:03.876909 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.876874    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 15:50:03.876909 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.876889    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 15:50:03.876909 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.876894    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 15:50:03.876909 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.876902    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 15:50:03.876909 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.876908    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 15:50:03.877140 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.876916    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 15:50:03.877140 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.876922    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 15:50:03.878608 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.878597    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 15:50:03.878642 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.878609    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 15:50:03.882553 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.882530    2574 server.go:1268] "Started kubelet"
Sep 16 15:50:03.882661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.882615    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 15:50:03.882738 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.882695    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 15:50:03.882795 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.882765    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 15:50:03.883502 ip-10-0-143-143 systemd[1]: Started Kubernetes Kubelet.
Sep 16 15:50:03.884496 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.884199    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 15:50:03.884852 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.884837    2574 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 15:50:03.888500 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:03.888471    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 15:50:03.891897 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.891880    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 15:50:03.892355 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.892337    2574 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-143.ec2.internal" not found
Sep 16 15:50:03.892536 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.892500    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 15:50:03.892911 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:03.892758    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-143.ec2.internal\" not found"
Sep 16 15:50:03.893342 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.893324    2574 factory.go:55] Registering systemd factory
Sep 16 15:50:03.893342 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.893344    2574 factory.go:223] Registration of the systemd container factory successfully
Sep 16 15:50:03.893465 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.893402    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 15:50:03.893465 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.893427    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 15:50:03.893465 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.893438    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 15:50:03.893590 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.893532    2574 factory.go:153] Registering CRI-O factory
Sep 16 15:50:03.893590 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.893542    2574 factory.go:223] Registration of the crio container factory successfully
Sep 16 15:50:03.893649 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.893603    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 15:50:03.893649 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.893619    2574 factory.go:103] Registering Raw factory
Sep 16 15:50:03.893649 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.893634    2574 manager.go:1196] Started watching for new ooms in manager
Sep 16 15:50:03.893991 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.893979    2574 manager.go:349] Starting recovery of all containers
Sep 16 15:50:03.894256 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.894244    2574 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 15:50:03.894289 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.894257    2574 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 15:50:03.894831 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.894776    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 15:50:03.901795 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:03.901770    2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-143.ec2.internal\" not found" node="ip-10-0-143-143.ec2.internal"
Sep 16 15:50:03.906653 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.906633    2574 manager.go:354] Recovery completed
Sep 16 15:50:03.907686 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.907670    2574 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-143.ec2.internal" not found
Sep 16 15:50:03.910874 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.910861    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 15:50:03.912631 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.912605    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 15:50:03.912702 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.912650    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 15:50:03.912702 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.912662    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasSufficientPID"
Sep 16 15:50:03.913278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.913262    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 15:50:03.913278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.913275    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 15:50:03.913378 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.913295    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 15:50:03.913378 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.913279    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 15:50:03.914008 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.913991    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 15:50:03.914081 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.914023    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 15:50:03.914081 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.914037    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasSufficientPID"
Sep 16 15:50:03.915569 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.915554    2574 policy_none.go:49] "None policy: Start"
Sep 16 15:50:03.915632 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.915574    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 15:50:03.915632 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.915588    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 15:50:03.917584 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.917570    2574 policy_none.go:47] "Start"
Sep 16 15:50:03.953170 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.953147    2574 manager.go:337] "Starting Device Plugin manager"
Sep 16 15:50:03.970832 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:03.953189    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 15:50:03.970832 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.953203    2574 server.go:87] "Starting device plugin registration server"
Sep 16 15:50:03.970832 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.953645    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 15:50:03.970832 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.953658    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 15:50:03.970832 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.953770    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 15:50:03.970832 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.953849    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 15:50:03.970832 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.953857    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 15:50:03.970832 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:03.954386    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 15:50:03.970832 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:03.954441    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-143.ec2.internal\" not found"
Sep 16 15:50:03.970832 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.964611    2574 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-143.ec2.internal" not found
Sep 16 15:50:03.986372 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.986331    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 15:50:03.987782 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.987754    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 15:50:03.987907 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.987821    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 15:50:03.987907 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.987852    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 15:50:03.987907 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:03.987900    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 15:50:03.990256 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:03.990237    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 15:50:04.054577 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.054471    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 15:50:04.055484 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.055461    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 15:50:04.055579 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.055494    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 15:50:04.055579 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.055504    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasSufficientPID"
Sep 16 15:50:04.055579 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.055568    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.062401 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.062386    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.062493 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:04.062408    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-143.ec2.internal\": node \"ip-10-0-143-143.ec2.internal\" not found"
Sep 16 15:50:04.071421 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:04.071401    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-143.ec2.internal\" not found"
Sep 16 15:50:04.088811 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.088783    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-143.ec2.internal"]
Sep 16 15:50:04.088881 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.088862    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 15:50:04.089843 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.089828    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 15:50:04.089898 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.089857    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 15:50:04.089898 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.089868    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasSufficientPID"
Sep 16 15:50:04.092330 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.092317    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 15:50:04.092479 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.092464    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.092512 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.092495    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 15:50:04.093107 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.093093    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 15:50:04.093168 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.093119    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 15:50:04.093168 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.093130    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasSufficientPID"
Sep 16 15:50:04.093967 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.093949    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 15:50:04.094049 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.093982    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 15:50:04.094049 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.093997    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasSufficientPID"
Sep 16 15:50:04.096131 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.096116    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/80e7377d29e102f842e6625acdbbaa55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal\" (UID: \"80e7377d29e102f842e6625acdbbaa55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.096190 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.096138    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80e7377d29e102f842e6625acdbbaa55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal\" (UID: \"80e7377d29e102f842e6625acdbbaa55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.096190 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.096180    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed97d765e2c9a80c45b1435ad72f4818-config\") pod \"kube-apiserver-proxy-ip-10-0-143-143.ec2.internal\" (UID: \"ed97d765e2c9a80c45b1435ad72f4818\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.096307 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.096293    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.096387 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.096318    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 15:50:04.096972 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.096956    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 15:50:04.097038 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.096981    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 15:50:04.097038 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.096991    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeHasSufficientPID"
Sep 16 15:50:04.116271 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:04.116249    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-143.ec2.internal\" not found" node="ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.120553 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:04.120538    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-143.ec2.internal\" not found" node="ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.172012 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:04.171973    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-143.ec2.internal\" not found"
Sep 16 15:50:04.197027 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.197000    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/80e7377d29e102f842e6625acdbbaa55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal\" (UID: \"80e7377d29e102f842e6625acdbbaa55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.197164 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.197032    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80e7377d29e102f842e6625acdbbaa55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal\" (UID: \"80e7377d29e102f842e6625acdbbaa55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.197164 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.197064    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed97d765e2c9a80c45b1435ad72f4818-config\") pod \"kube-apiserver-proxy-ip-10-0-143-143.ec2.internal\" (UID: \"ed97d765e2c9a80c45b1435ad72f4818\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.197164 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.197103    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/80e7377d29e102f842e6625acdbbaa55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal\" (UID: \"80e7377d29e102f842e6625acdbbaa55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.197164 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.197115    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed97d765e2c9a80c45b1435ad72f4818-config\") pod \"kube-apiserver-proxy-ip-10-0-143-143.ec2.internal\" (UID: \"ed97d765e2c9a80c45b1435ad72f4818\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.197164 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.197116    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80e7377d29e102f842e6625acdbbaa55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal\" (UID: \"80e7377d29e102f842e6625acdbbaa55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.272765 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:04.272715    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-143.ec2.internal\" not found"
Sep 16 15:50:04.373547 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:04.373490    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-143.ec2.internal\" not found"
Sep 16 15:50:04.419580 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.419542    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.422927 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.422900    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:04.475280 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:04.475247    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-143.ec2.internal\" not found"
Sep 16 15:50:04.579985 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:04.579949    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-143.ec2.internal\" not found"
Sep 16 15:50:04.680437 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:04.680341    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-143.ec2.internal\" not found"
Sep 16 15:50:04.759893 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.759861    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 15:50:04.781909 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:04.781871    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-143.ec2.internal\" not found"
Sep 16 15:50:04.816669 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.816640    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 15:50:04.817142 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.816820    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 15:50:04.817142 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.816869    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 15:50:04.817142 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.816899    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 15:50:04.879647 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.879566    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 15:45:04 +0000 UTC" deadline="2028-02-19 13:10:47.185504307 +0000 UTC"
Sep 16 15:50:04.879647 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.879627    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12501h20m42.305881085s"
Sep 16 15:50:04.882741 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:04.882711    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-143.ec2.internal\" not found"
Sep 16 15:50:04.971744 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.971675    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 15:50:04.993634 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:04.993605    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:05.004685 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.004664    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 15:50:05.005869 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.005854    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-143.ec2.internal"
Sep 16 15:50:05.014352 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.014329    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 15:50:05.030297 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:05.030265    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded97d765e2c9a80c45b1435ad72f4818.slice/crio-284c3f4861920adf5f8b03eca7938958391dfcd3f208c81881603bb6cf01dc34 WatchSource:0}: Error finding container 284c3f4861920adf5f8b03eca7938958391dfcd3f208c81881603bb6cf01dc34: Status 404 returned error can't find the container with id 284c3f4861920adf5f8b03eca7938958391dfcd3f208c81881603bb6cf01dc34
Sep 16 15:50:05.031080 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:05.031051    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e7377d29e102f842e6625acdbbaa55.slice/crio-ab1fdfec92c75eaef761fb9be592980aeb72a8252fd9a9c141a44f48f943ce6d WatchSource:0}: Error finding container ab1fdfec92c75eaef761fb9be592980aeb72a8252fd9a9c141a44f48f943ce6d: Status 404 returned error can't find the container with id ab1fdfec92c75eaef761fb9be592980aeb72a8252fd9a9c141a44f48f943ce6d
Sep 16 15:50:05.035353 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.035334    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 15:50:05.660717 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.660680    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 15:50:05.867316 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.867282    2574 apiserver.go:52] "Watching apiserver"
Sep 16 15:50:05.874424 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.874392    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 15:50:05.876166 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.876087    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-7knnk","kube-system/global-pull-secret-syncer-86l22","openshift-cluster-node-tuning-operator/tuned-6wnxz","openshift-image-registry/node-ca-nz7nc","openshift-multus/multus-7jbf9","openshift-network-operator/iptables-alerter-nt24x","openshift-ovn-kubernetes/ovnkube-node-ldr94","kube-system/konnectivity-agent-s2k52","kube-system/kube-apiserver-proxy-ip-10-0-143-143.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9","openshift-dns/node-resolver-6r28d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal","openshift-multus/multus-additional-cni-plugins-2w8vv","openshift-multus/network-metrics-daemon-bbfmm"]
Sep 16 15:50:05.879127 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.879100    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.881279 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.881249    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tmhl\""
Sep 16 15:50:05.881633 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.881618    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:05.882251 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.882225    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 15:50:05.882349 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.882316    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 15:50:05.883332 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.883308    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 15:50:05.883438 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.883370    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 15:50:05.883500 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.883472    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 15:50:05.883571 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.883542    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 15:50:05.884207 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.884188    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.884717 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.884689    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-748rb\""
Sep 16 15:50:05.884816 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.884741    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 15:50:05.884816 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.884751    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 15:50:05.885059 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.885044    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 15:50:05.885238 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.885224    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 15:50:05.885529 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.885399    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 15:50:05.886026 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.886005    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n6dwg\""
Sep 16 15:50:05.886147 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.886031    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 15:50:05.886147 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.886049    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 15:50:05.887006 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.886754    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nz7nc"
Sep 16 15:50:05.888635 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.888596    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 15:50:05.888824 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.888804    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 15:50:05.888960 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.888855    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 15:50:05.888960 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.888858    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xjcsb\""
Sep 16 15:50:05.889278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.889260    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.891998 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.891978    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 15:50:05.892105 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.892095    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nt24x"
Sep 16 15:50:05.892193 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.892175    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 15:50:05.892284 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.892263    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9dmzp\""
Sep 16 15:50:05.894236 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.894216    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 15:50:05.894630 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.894612    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:05.894718 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:05.894686    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:05.894861 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.894840    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 15:50:05.894969 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.894952    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 15:50:05.895289 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.895271    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fj2tn\""
Sep 16 15:50:05.897308 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.897285    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:05.897414 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:05.897360    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:05.899820 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.899789    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:05.899919 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:05.899869    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:05.902803 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.902785    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:05.904819 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.904801    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6r28d"
Sep 16 15:50:05.907346 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.907305    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s2k52"
Sep 16 15:50:05.907643 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.907622    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:05.907751 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.907662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz4pf\" (UniqueName: \"kubernetes.io/projected/5b155b34-181a-43f2-a2e7-a4c540287735-kube-api-access-hz4pf\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.907751 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.907732    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12c61b9c-b819-4330-bcf8-05341ce39ac8-serviceca\") pod \"node-ca-nz7nc\" (UID: \"12c61b9c-b819-4330-bcf8-05341ce39ac8\") " pod="openshift-image-registry/node-ca-nz7nc"
Sep 16 15:50:05.907867 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.907762    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-var-lib-kubelet\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.907867 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.907790    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29sc6\" (UniqueName: \"kubernetes.io/projected/46cd7e46-0254-4c9c-b274-136e18ba7d91-kube-api-access-29sc6\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.907867 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.907819    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-modprobe-d\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.907867 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.907845    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-var-lib-cni-bin\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.908123 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.907871    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-run-openvswitch\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.908123 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.907896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpgv\" (UniqueName: \"kubernetes.io/projected/12c61b9c-b819-4330-bcf8-05341ce39ac8-kube-api-access-xkpgv\") pod \"node-ca-nz7nc\" (UID: \"12c61b9c-b819-4330-bcf8-05341ce39ac8\") " pod="openshift-image-registry/node-ca-nz7nc"
Sep 16 15:50:05.908123 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.907924    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-multus-cni-dir\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.908123 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.907947    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-run-netns\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.908123 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.907972    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45xx\" (UniqueName: \"kubernetes.io/projected/c685b2f9-18ae-4c7f-b9c7-8f271c676488-kube-api-access-g45xx\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.908123 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908027    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe3174a1-8fb2-402d-9826-df2eb068dfcb-iptables-alerter-script\") pod \"iptables-alerter-nt24x\" (UID: \"fe3174a1-8fb2-402d-9826-df2eb068dfcb\") " pod="openshift-network-operator/iptables-alerter-nt24x"
Sep 16 15:50:05.908123 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908063    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-etc-openvswitch\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.908123 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908089    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.908123 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908113    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b155b34-181a-43f2-a2e7-a4c540287735-ovnkube-config\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.908123 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908148    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c685b2f9-18ae-4c7f-b9c7-8f271c676488-cni-binary-copy\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908184    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-var-lib-cni-multus\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908212    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908227    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-kubernetes\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908243    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-sysctl-conf\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908269    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-systemd\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908301    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-httxz\" (UniqueName: \"kubernetes.io/projected/fe3174a1-8fb2-402d-9826-df2eb068dfcb-kube-api-access-httxz\") pod \"iptables-alerter-nt24x\" (UID: \"fe3174a1-8fb2-402d-9826-df2eb068dfcb\") " pod="openshift-network-operator/iptables-alerter-nt24x"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908337    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12c61b9c-b819-4330-bcf8-05341ce39ac8-host\") pod \"node-ca-nz7nc\" (UID: \"12c61b9c-b819-4330-bcf8-05341ce39ac8\") " pod="openshift-image-registry/node-ca-nz7nc"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908371    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-sysctl-d\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908386    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-hostroot\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908401    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-etc-kubernetes\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908425    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-cnibin\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908444    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908461    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-run-multus-certs\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908477    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-var-lib-openvswitch\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.908710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908514    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-host\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908607    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46cd7e46-0254-4c9c-b274-136e18ba7d91-tmp\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908631    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-systemd-units\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908651    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-tuned\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908672    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-system-cni-dir\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908695    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-cnibin\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-slash\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908752    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-run-netns\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908787    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908823    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-cni-bin\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908852    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-cni-netd\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908874    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-run\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908895    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-sysconfig\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908933    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-multus-conf-dir\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908967    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b155b34-181a-43f2-a2e7-a4c540287735-env-overrides\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.908982    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-lib-modules\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909000    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe3174a1-8fb2-402d-9826-df2eb068dfcb-host-slash\") pod \"iptables-alerter-nt24x\" (UID: \"fe3174a1-8fb2-402d-9826-df2eb068dfcb\") " pod="openshift-network-operator/iptables-alerter-nt24x"
Sep 16 15:50:05.909610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909025    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-system-cni-dir\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909058    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c685b2f9-18ae-4c7f-b9c7-8f271c676488-multus-daemon-config\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909095    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-run-systemd\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909126    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzj4g\" (UniqueName: \"kubernetes.io/projected/1332ae3f-698f-4d0e-9801-2aa87121a12b-kube-api-access-lzj4g\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909163    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-run-ovn\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909210    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-log-socket\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909234    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b155b34-181a-43f2-a2e7-a4c540287735-ovnkube-script-lib\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909259    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909286    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-run-k8s-cni-cncf-io\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909310    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-var-lib-kubelet\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909334    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-node-log\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909369    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909359    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b155b34-181a-43f2-a2e7-a4c540287735-ovn-node-metrics-cert\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909387    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909412    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-sys\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909437    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-multus-socket-dir-parent\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909459    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-kubelet\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909482    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9sbl\" (UniqueName: \"kubernetes.io/projected/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-kube-api-access-k9sbl\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:05.910509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909646    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 15:50:05.911363 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909681    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 15:50:05.911363 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909727    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rs9mr\""
Sep 16 15:50:05.911363 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.909895    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zdsmg\""
Sep 16 15:50:05.911363 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.910106    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 15:50:05.911363 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.910559    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fmjgz\""
Sep 16 15:50:05.911363 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.910750    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 15:50:05.911788 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.911473    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 15:50:05.925097 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.925052    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 15:50:05.966856 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.966830    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2khkz"
Sep 16 15:50:05.975522 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.975494    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2khkz"
Sep 16 15:50:05.992755 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.992703    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal" event={"ID":"80e7377d29e102f842e6625acdbbaa55","Type":"ContainerStarted","Data":"ab1fdfec92c75eaef761fb9be592980aeb72a8252fd9a9c141a44f48f943ce6d"}
Sep 16 15:50:05.993892 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.993860    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-143.ec2.internal" event={"ID":"ed97d765e2c9a80c45b1435ad72f4818","Type":"ContainerStarted","Data":"284c3f4861920adf5f8b03eca7938958391dfcd3f208c81881603bb6cf01dc34"}
Sep 16 15:50:05.994287 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:05.994268    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 15:50:06.010081 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010048    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-cni-bin\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.010248 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010089    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-cni-netd\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.010248 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010115    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-run\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.010248 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010165    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-cni-bin\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.010248 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010173    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-cni-netd\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.010248 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010175    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-run\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.010248 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010194    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-sysconfig\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.010248 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010233    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-sysconfig\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.010661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-multus-conf-dir\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.010661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010290    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b155b34-181a-43f2-a2e7-a4c540287735-env-overrides\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.010661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010317    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-lib-modules\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.010661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010348    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-socket-dir\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.010661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010373    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-registration-dir\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.010661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010395    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9823bb5e-f7d2-442d-b83b-c833dafb42ba-dbus\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:06.010661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010419    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe3174a1-8fb2-402d-9826-df2eb068dfcb-host-slash\") pod \"iptables-alerter-nt24x\" (UID: \"fe3174a1-8fb2-402d-9826-df2eb068dfcb\") " pod="openshift-network-operator/iptables-alerter-nt24x"
Sep 16 15:50:06.010661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010443    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-system-cni-dir\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.010661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010348    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-multus-conf-dir\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.010661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a690936-8677-451d-8ff9-c6b3b92d666a-konnectivity-ca\") pod \"konnectivity-agent-s2k52\" (UID: \"9a690936-8677-451d-8ff9-c6b3b92d666a\") " pod="kube-system/konnectivity-agent-s2k52"
Sep 16 15:50:06.010661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010506    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-lib-modules\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.010661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010535    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe3174a1-8fb2-402d-9826-df2eb068dfcb-host-slash\") pod \"iptables-alerter-nt24x\" (UID: \"fe3174a1-8fb2-402d-9826-df2eb068dfcb\") " pod="openshift-network-operator/iptables-alerter-nt24x"
Sep 16 15:50:06.010661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010556    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c685b2f9-18ae-4c7f-b9c7-8f271c676488-multus-daemon-config\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.010661 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010564    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-system-cni-dir\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010689    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-run-systemd\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010758    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-run-systemd\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010777    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27cn4\" (UniqueName: \"kubernetes.io/projected/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-kube-api-access-27cn4\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010820    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eedc6ccd-a007-4ca4-98ba-54fa62367075-hosts-file\") pod \"node-resolver-6r28d\" (UID: \"eedc6ccd-a007-4ca4-98ba-54fa62367075\") " pod="openshift-dns/node-resolver-6r28d"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010842    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b155b34-181a-43f2-a2e7-a4c540287735-env-overrides\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010859    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-etc-selinux\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010886    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-sys-fs\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010919    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzj4g\" (UniqueName: \"kubernetes.io/projected/1332ae3f-698f-4d0e-9801-2aa87121a12b-kube-api-access-lzj4g\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010947    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-run-ovn\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010972    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-log-socket\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.010998    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b155b34-181a-43f2-a2e7-a4c540287735-ovnkube-script-lib\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011022    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-run-ovn\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011025    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011058    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-run-k8s-cni-cncf-io\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011080    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-var-lib-kubelet\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011096    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-log-socket\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.011338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011110    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-var-lib-kubelet\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011149    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c685b2f9-18ae-4c7f-b9c7-8f271c676488-multus-daemon-config\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011149    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-node-log\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011172    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-run-k8s-cni-cncf-io\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011187    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-node-log\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011203    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b155b34-181a-43f2-a2e7-a4c540287735-ovn-node-metrics-cert\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011172    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-sys\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011278    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7sf2\" (UniqueName: \"kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2\") pod \"network-check-target-7knnk\" (UID: \"76cadf23-c5f4-4d75-98ab-48509fd8cb42\") " pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011307    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-device-dir\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011336    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-multus-socket-dir-parent\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011342    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-sys\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011389    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-kubelet\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011414    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-multus-socket-dir-parent\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011419    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9sbl\" (UniqueName: \"kubernetes.io/projected/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-kube-api-access-k9sbl\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011459    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-kubelet\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.012225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011569    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b155b34-181a-43f2-a2e7-a4c540287735-ovnkube-script-lib\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011593    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz4pf\" (UniqueName: \"kubernetes.io/projected/5b155b34-181a-43f2-a2e7-a4c540287735-kube-api-access-hz4pf\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011622    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12c61b9c-b819-4330-bcf8-05341ce39ac8-serviceca\") pod \"node-ca-nz7nc\" (UID: \"12c61b9c-b819-4330-bcf8-05341ce39ac8\") " pod="openshift-image-registry/node-ca-nz7nc"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011647    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-var-lib-kubelet\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011672    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29sc6\" (UniqueName: \"kubernetes.io/projected/46cd7e46-0254-4c9c-b274-136e18ba7d91-kube-api-access-29sc6\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011698    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011724    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-modprobe-d\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.011750    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011769    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-var-lib-cni-bin\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-run-openvswitch\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011722    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-var-lib-kubelet\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011848    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-run-openvswitch\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.011856    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs podName:1332ae3f-698f-4d0e-9801-2aa87121a12b nodeName:}" failed. No retries permitted until 2026-09-16 15:50:06.511810312 +0000 UTC m=+3.031433086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs") pod "network-metrics-daemon-bbfmm" (UID: "1332ae3f-698f-4d0e-9801-2aa87121a12b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpgv\" (UniqueName: \"kubernetes.io/projected/12c61b9c-b819-4330-bcf8-05341ce39ac8-kube-api-access-xkpgv\") pod \"node-ca-nz7nc\" (UID: \"12c61b9c-b819-4330-bcf8-05341ce39ac8\") " pod="openshift-image-registry/node-ca-nz7nc"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011896    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-var-lib-cni-bin\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011928    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-multus-cni-dir\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.013101 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011927    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-modprobe-d\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011979    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-run-netns\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011992    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45xx\" (UniqueName: \"kubernetes.io/projected/c685b2f9-18ae-4c7f-b9c7-8f271c676488-kube-api-access-g45xx\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.011991    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-multus-cni-dir\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012041    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe3174a1-8fb2-402d-9826-df2eb068dfcb-iptables-alerter-script\") pod \"iptables-alerter-nt24x\" (UID: \"fe3174a1-8fb2-402d-9826-df2eb068dfcb\") " pod="openshift-network-operator/iptables-alerter-nt24x"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012070    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-run-netns\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012086    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-etc-openvswitch\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012114    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012142    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b155b34-181a-43f2-a2e7-a4c540287735-ovnkube-config\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012168    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012174    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-etc-openvswitch\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012196    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c685b2f9-18ae-4c7f-b9c7-8f271c676488-cni-binary-copy\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012223    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-var-lib-cni-multus\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012250    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012276    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-kubernetes\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012303    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-sysctl-conf\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.013984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012390    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012327    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-systemd\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a690936-8677-451d-8ff9-c6b3b92d666a-agent-certs\") pod \"konnectivity-agent-s2k52\" (UID: \"9a690936-8677-451d-8ff9-c6b3b92d666a\") " pod="kube-system/konnectivity-agent-s2k52"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012884    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12c61b9c-b819-4330-bcf8-05341ce39ac8-serviceca\") pod \"node-ca-nz7nc\" (UID: \"12c61b9c-b819-4330-bcf8-05341ce39ac8\") " pod="openshift-image-registry/node-ca-nz7nc"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012918    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httxz\" (UniqueName: \"kubernetes.io/projected/fe3174a1-8fb2-402d-9826-df2eb068dfcb-kube-api-access-httxz\") pod \"iptables-alerter-nt24x\" (UID: \"fe3174a1-8fb2-402d-9826-df2eb068dfcb\") " pod="openshift-network-operator/iptables-alerter-nt24x"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012945    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12c61b9c-b819-4330-bcf8-05341ce39ac8-host\") pod \"node-ca-nz7nc\" (UID: \"12c61b9c-b819-4330-bcf8-05341ce39ac8\") " pod="openshift-image-registry/node-ca-nz7nc"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012950    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012971    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-sysctl-d\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.012996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-hostroot\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013013    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-etc-kubernetes\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013012    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-var-lib-cni-multus\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013028    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-cnibin\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013076    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9823bb5e-f7d2-442d-b83b-c833dafb42ba-kubelet-config\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013102    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flrnm\" (UniqueName: \"kubernetes.io/projected/eedc6ccd-a007-4ca4-98ba-54fa62367075-kube-api-access-flrnm\") pod \"node-resolver-6r28d\" (UID: \"eedc6ccd-a007-4ca4-98ba-54fa62367075\") " pod="openshift-dns/node-resolver-6r28d"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013110    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-sysctl-conf\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013115    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-sysctl-d\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.014733 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013130    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-run-multus-certs\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-var-lib-openvswitch\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013167    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12c61b9c-b819-4330-bcf8-05341ce39ac8-host\") pod \"node-ca-nz7nc\" (UID: \"12c61b9c-b819-4330-bcf8-05341ce39ac8\") " pod="openshift-image-registry/node-ca-nz7nc"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013171    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-systemd\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013165    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c685b2f9-18ae-4c7f-b9c7-8f271c676488-cni-binary-copy\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013025    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-kubernetes\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013186    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-host\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013225    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-hostroot\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013226    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-etc-kubernetes\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46cd7e46-0254-4c9c-b274-136e18ba7d91-tmp\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-systemd-units\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013319    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-tuned\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013341    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eedc6ccd-a007-4ca4-98ba-54fa62367075-tmp-dir\") pod \"node-resolver-6r28d\" (UID: \"eedc6ccd-a007-4ca4-98ba-54fa62367075\") " pod="openshift-dns/node-resolver-6r28d"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013364    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-system-cni-dir\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013386    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-cnibin\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013455    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-slash\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013478    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-run-netns\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013482    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-host-run-multus-certs\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.015436 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013500    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.016303 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013536    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-cnibin\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.016303 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013573    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.016303 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013578    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.016303 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013609    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46cd7e46-0254-4c9c-b274-136e18ba7d91-host\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.016303 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013638    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-cnibin\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.016303 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013642    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-var-lib-openvswitch\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.016303 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013675    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c685b2f9-18ae-4c7f-b9c7-8f271c676488-system-cni-dir\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.016303 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013686    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-systemd-units\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.016303 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013712    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-slash\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.016303 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013721    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.016303 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.013755    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b155b34-181a-43f2-a2e7-a4c540287735-host-run-netns\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.016303 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.014163    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b155b34-181a-43f2-a2e7-a4c540287735-ovnkube-config\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.016303 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.014211    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fe3174a1-8fb2-402d-9826-df2eb068dfcb-iptables-alerter-script\") pod \"iptables-alerter-nt24x\" (UID: \"fe3174a1-8fb2-402d-9826-df2eb068dfcb\") " pod="openshift-network-operator/iptables-alerter-nt24x"
Sep 16 15:50:06.016899 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.016845    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46cd7e46-0254-4c9c-b274-136e18ba7d91-tmp\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.016938 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.016913    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46cd7e46-0254-4c9c-b274-136e18ba7d91-etc-tuned\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.017110 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.017091    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b155b34-181a-43f2-a2e7-a4c540287735-ovn-node-metrics-cert\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.026248 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.026222    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpgv\" (UniqueName: \"kubernetes.io/projected/12c61b9c-b819-4330-bcf8-05341ce39ac8-kube-api-access-xkpgv\") pod \"node-ca-nz7nc\" (UID: \"12c61b9c-b819-4330-bcf8-05341ce39ac8\") " pod="openshift-image-registry/node-ca-nz7nc"
Sep 16 15:50:06.026386 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.026252    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29sc6\" (UniqueName: \"kubernetes.io/projected/46cd7e46-0254-4c9c-b274-136e18ba7d91-kube-api-access-29sc6\") pod \"tuned-6wnxz\" (UID: \"46cd7e46-0254-4c9c-b274-136e18ba7d91\") " pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.026982 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.026934    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-httxz\" (UniqueName: \"kubernetes.io/projected/fe3174a1-8fb2-402d-9826-df2eb068dfcb-kube-api-access-httxz\") pod \"iptables-alerter-nt24x\" (UID: \"fe3174a1-8fb2-402d-9826-df2eb068dfcb\") " pod="openshift-network-operator/iptables-alerter-nt24x"
Sep 16 15:50:06.027076 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.027035    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzj4g\" (UniqueName: \"kubernetes.io/projected/1332ae3f-698f-4d0e-9801-2aa87121a12b-kube-api-access-lzj4g\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:06.027927 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.027900    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9sbl\" (UniqueName: \"kubernetes.io/projected/9fe8739d-6acf-41c1-b1c8-f11a8fa361ca-kube-api-access-k9sbl\") pod \"multus-additional-cni-plugins-2w8vv\" (UID: \"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca\") " pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.028132 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.028102    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45xx\" (UniqueName: \"kubernetes.io/projected/c685b2f9-18ae-4c7f-b9c7-8f271c676488-kube-api-access-g45xx\") pod \"multus-7jbf9\" (UID: \"c685b2f9-18ae-4c7f-b9c7-8f271c676488\") " pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.028299 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.028281    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz4pf\" (UniqueName: \"kubernetes.io/projected/5b155b34-181a-43f2-a2e7-a4c540287735-kube-api-access-hz4pf\") pod \"ovnkube-node-ldr94\" (UID: \"5b155b34-181a-43f2-a2e7-a4c540287735\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.113896 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.113852    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-etc-selinux\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.113896 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.113898    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-sys-fs\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.114151 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.113926    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7sf2\" (UniqueName: \"kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2\") pod \"network-check-target-7knnk\" (UID: \"76cadf23-c5f4-4d75-98ab-48509fd8cb42\") " pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:06.114151 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.113954    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-device-dir\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.114151 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.113982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.114151 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114063    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.114151 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114063    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-etc-selinux\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.114151 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114121    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-device-dir\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.114151 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114131    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:06.114151 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114126    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-sys-fs\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.114545 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114205    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a690936-8677-451d-8ff9-c6b3b92d666a-agent-certs\") pod \"konnectivity-agent-s2k52\" (UID: \"9a690936-8677-451d-8ff9-c6b3b92d666a\") " pod="kube-system/konnectivity-agent-s2k52"
Sep 16 15:50:06.114545 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114237    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9823bb5e-f7d2-442d-b83b-c833dafb42ba-kubelet-config\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:06.114545 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114262    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flrnm\" (UniqueName: \"kubernetes.io/projected/eedc6ccd-a007-4ca4-98ba-54fa62367075-kube-api-access-flrnm\") pod \"node-resolver-6r28d\" (UID: \"eedc6ccd-a007-4ca4-98ba-54fa62367075\") " pod="openshift-dns/node-resolver-6r28d"
Sep 16 15:50:06.114545 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114299    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eedc6ccd-a007-4ca4-98ba-54fa62367075-tmp-dir\") pod \"node-resolver-6r28d\" (UID: \"eedc6ccd-a007-4ca4-98ba-54fa62367075\") " pod="openshift-dns/node-resolver-6r28d"
Sep 16 15:50:06.114545 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.114308    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:06.114545 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114351    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9823bb5e-f7d2-442d-b83b-c833dafb42ba-kubelet-config\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:06.114545 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.114394    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret podName:9823bb5e-f7d2-442d-b83b-c833dafb42ba nodeName:}" failed. No retries permitted until 2026-09-16 15:50:06.614368446 +0000 UTC m=+3.133991214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret") pod "global-pull-secret-syncer-86l22" (UID: "9823bb5e-f7d2-442d-b83b-c833dafb42ba") : object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:06.114545 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114425    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-socket-dir\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.114545 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114457    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-registration-dir\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.114545 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114483    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9823bb5e-f7d2-442d-b83b-c833dafb42ba-dbus\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:06.114545 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a690936-8677-451d-8ff9-c6b3b92d666a-konnectivity-ca\") pod \"konnectivity-agent-s2k52\" (UID: \"9a690936-8677-451d-8ff9-c6b3b92d666a\") " pod="kube-system/konnectivity-agent-s2k52"
Sep 16 15:50:06.114545 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114521    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27cn4\" (UniqueName: \"kubernetes.io/projected/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-kube-api-access-27cn4\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.114545 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114554    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eedc6ccd-a007-4ca4-98ba-54fa62367075-hosts-file\") pod \"node-resolver-6r28d\" (UID: \"eedc6ccd-a007-4ca4-98ba-54fa62367075\") " pod="openshift-dns/node-resolver-6r28d"
Sep 16 15:50:06.115118 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114571    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-socket-dir\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.115118 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114608    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9823bb5e-f7d2-442d-b83b-c833dafb42ba-dbus\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:06.115118 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114644    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eedc6ccd-a007-4ca4-98ba-54fa62367075-tmp-dir\") pod \"node-resolver-6r28d\" (UID: \"eedc6ccd-a007-4ca4-98ba-54fa62367075\") " pod="openshift-dns/node-resolver-6r28d"
Sep 16 15:50:06.115118 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114652    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eedc6ccd-a007-4ca4-98ba-54fa62367075-hosts-file\") pod \"node-resolver-6r28d\" (UID: \"eedc6ccd-a007-4ca4-98ba-54fa62367075\") " pod="openshift-dns/node-resolver-6r28d"
Sep 16 15:50:06.115118 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.114670    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-registration-dir\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.115372 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.115257    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a690936-8677-451d-8ff9-c6b3b92d666a-konnectivity-ca\") pod \"konnectivity-agent-s2k52\" (UID: \"9a690936-8677-451d-8ff9-c6b3b92d666a\") " pod="kube-system/konnectivity-agent-s2k52"
Sep 16 15:50:06.117064 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.117013    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a690936-8677-451d-8ff9-c6b3b92d666a-agent-certs\") pod \"konnectivity-agent-s2k52\" (UID: \"9a690936-8677-451d-8ff9-c6b3b92d666a\") " pod="kube-system/konnectivity-agent-s2k52"
Sep 16 15:50:06.120801 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.120731    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 15:50:06.120801 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.120760    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 15:50:06.120801 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.120777    2574 projected.go:196] Error preparing data for projected volume kube-api-access-z7sf2 for pod openshift-network-diagnostics/network-check-target-7knnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:06.121039 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.120873    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2 podName:76cadf23-c5f4-4d75-98ab-48509fd8cb42 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:06.620849476 +0000 UTC m=+3.140472254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z7sf2" (UniqueName: "kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2") pod "network-check-target-7knnk" (UID: "76cadf23-c5f4-4d75-98ab-48509fd8cb42") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:06.122935 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.122912    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flrnm\" (UniqueName: \"kubernetes.io/projected/eedc6ccd-a007-4ca4-98ba-54fa62367075-kube-api-access-flrnm\") pod \"node-resolver-6r28d\" (UID: \"eedc6ccd-a007-4ca4-98ba-54fa62367075\") " pod="openshift-dns/node-resolver-6r28d"
Sep 16 15:50:06.123458 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.123430    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27cn4\" (UniqueName: \"kubernetes.io/projected/1a7785e3-0b97-40ee-8fb8-b4c767aadf11-kube-api-access-27cn4\") pod \"aws-ebs-csi-driver-node-bc8r9\" (UID: \"1a7785e3-0b97-40ee-8fb8-b4c767aadf11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.194446 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.194356    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:06.202412 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.202390    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2w8vv"
Sep 16 15:50:06.212449 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.212424    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6wnxz"
Sep 16 15:50:06.220687 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.220665    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nz7nc"
Sep 16 15:50:06.228662 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.228638    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7jbf9"
Sep 16 15:50:06.236644 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.236611    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nt24x"
Sep 16 15:50:06.244660 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.244635    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9"
Sep 16 15:50:06.253699 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.253659    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6r28d"
Sep 16 15:50:06.259639 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.259611    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s2k52"
Sep 16 15:50:06.305820 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.305764    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 15:50:06.336788 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:06.336324    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedc6ccd_a007_4ca4_98ba_54fa62367075.slice/crio-17e3cd2b1886f451bd378076273c9877fd51f7bf7fe208b879d79f23161b0ab7 WatchSource:0}: Error finding container 17e3cd2b1886f451bd378076273c9877fd51f7bf7fe208b879d79f23161b0ab7: Status 404 returned error can't find the container with id 17e3cd2b1886f451bd378076273c9877fd51f7bf7fe208b879d79f23161b0ab7
Sep 16 15:50:06.340327 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:06.340296    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a690936_8677_451d_8ff9_c6b3b92d666a.slice/crio-44dadb38a388c146e4955a8c6e9f789430769496847c17dde5198c7e8a99a77b WatchSource:0}: Error finding container 44dadb38a388c146e4955a8c6e9f789430769496847c17dde5198c7e8a99a77b: Status 404 returned error can't find the container with id 44dadb38a388c146e4955a8c6e9f789430769496847c17dde5198c7e8a99a77b
Sep 16 15:50:06.340998 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:06.340976    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7785e3_0b97_40ee_8fb8_b4c767aadf11.slice/crio-21a998e07cb6eb4c95475dbac8f9caee5ee66a7349e43aaeed43f11856ab44ec WatchSource:0}: Error finding container 21a998e07cb6eb4c95475dbac8f9caee5ee66a7349e43aaeed43f11856ab44ec: Status 404 returned error can't find the container with id 21a998e07cb6eb4c95475dbac8f9caee5ee66a7349e43aaeed43f11856ab44ec
Sep 16 15:50:06.341782 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:06.341763    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b155b34_181a_43f2_a2e7_a4c540287735.slice/crio-8c708af616c14219d9700a42b9e9a2001985e1560b4cf0fa9e794f50747703ac WatchSource:0}: Error finding container 8c708af616c14219d9700a42b9e9a2001985e1560b4cf0fa9e794f50747703ac: Status 404 returned error can't find the container with id 8c708af616c14219d9700a42b9e9a2001985e1560b4cf0fa9e794f50747703ac
Sep 16 15:50:06.346510 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:06.346395    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c61b9c_b819_4330_bcf8_05341ce39ac8.slice/crio-9e8c378455d38be529963d29109c74007bb1481aa35d097b3f51af892c727dc2 WatchSource:0}: Error finding container 9e8c378455d38be529963d29109c74007bb1481aa35d097b3f51af892c727dc2: Status 404 returned error can't find the container with id 9e8c378455d38be529963d29109c74007bb1481aa35d097b3f51af892c727dc2
Sep 16 15:50:06.347528 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:06.347361    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3174a1_8fb2_402d_9826_df2eb068dfcb.slice/crio-09528dc36f09bb8682c62c08f59cfc051346ea6806563218767e22ae3f28b0c5 WatchSource:0}: Error finding container 09528dc36f09bb8682c62c08f59cfc051346ea6806563218767e22ae3f28b0c5: Status 404 returned error can't find the container with id 09528dc36f09bb8682c62c08f59cfc051346ea6806563218767e22ae3f28b0c5
Sep 16 15:50:06.348622 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:06.348515    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe8739d_6acf_41c1_b1c8_f11a8fa361ca.slice/crio-11f21693a598c405aeedf0ff2b4d525a06fa582ac27f2632be24bcaa1490ff8c WatchSource:0}: Error finding container 11f21693a598c405aeedf0ff2b4d525a06fa582ac27f2632be24bcaa1490ff8c: Status 404 returned error can't find the container with id 11f21693a598c405aeedf0ff2b4d525a06fa582ac27f2632be24bcaa1490ff8c
Sep 16 15:50:06.517641 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.517538    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:06.517786 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.517711    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:06.517830 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.517789    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs podName:1332ae3f-698f-4d0e-9801-2aa87121a12b nodeName:}" failed. No retries permitted until 2026-09-16 15:50:07.517765717 +0000 UTC m=+4.037388491 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs") pod "network-metrics-daemon-bbfmm" (UID: "1332ae3f-698f-4d0e-9801-2aa87121a12b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:06.618635 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.618600    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:06.618796 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.618737    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:06.618850 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.618810    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret podName:9823bb5e-f7d2-442d-b83b-c833dafb42ba nodeName:}" failed. No retries permitted until 2026-09-16 15:50:07.618792323 +0000 UTC m=+4.138415094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret") pod "global-pull-secret-syncer-86l22" (UID: "9823bb5e-f7d2-442d-b83b-c833dafb42ba") : object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:06.719813 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.719769    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7sf2\" (UniqueName: \"kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2\") pod \"network-check-target-7knnk\" (UID: \"76cadf23-c5f4-4d75-98ab-48509fd8cb42\") " pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:06.719990 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.719965    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 15:50:06.720061 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.720025    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 15:50:06.720061 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.720043    2574 projected.go:196] Error preparing data for projected volume kube-api-access-z7sf2 for pod openshift-network-diagnostics/network-check-target-7knnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:06.720155 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.720136    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2 podName:76cadf23-c5f4-4d75-98ab-48509fd8cb42 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:07.720115641 +0000 UTC m=+4.239738410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7sf2" (UniqueName: "kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2") pod "network-check-target-7knnk" (UID: "76cadf23-c5f4-4d75-98ab-48509fd8cb42") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:06.922394 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.922032    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 15:50:06.978809 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.978706    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 15:45:06 +0000 UTC" deadline="2028-03-14 05:18:01.883924274 +0000 UTC"
Sep 16 15:50:06.978809 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.978749    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13069h27m54.905180633s"
Sep 16 15:50:06.989147 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:06.989084    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:06.989347 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:06.989267    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:07.000154 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.000060    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6wnxz" event={"ID":"46cd7e46-0254-4c9c-b274-136e18ba7d91","Type":"ContainerStarted","Data":"abccbb941db40b377320dfba75dfe81c4d8d9484433b3ab5fe181481955b16c3"}
Sep 16 15:50:07.001888 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.001782    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94" event={"ID":"5b155b34-181a-43f2-a2e7-a4c540287735","Type":"ContainerStarted","Data":"8c708af616c14219d9700a42b9e9a2001985e1560b4cf0fa9e794f50747703ac"}
Sep 16 15:50:07.007053 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.006987    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6r28d" event={"ID":"eedc6ccd-a007-4ca4-98ba-54fa62367075","Type":"ContainerStarted","Data":"17e3cd2b1886f451bd378076273c9877fd51f7bf7fe208b879d79f23161b0ab7"}
Sep 16 15:50:07.023893 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.020271    2574 generic.go:356] "Generic (PLEG): container finished" podID="80e7377d29e102f842e6625acdbbaa55" containerID="fde308fa7489fc916edadaf5b74cd8f1671eee7adb7214b92dc8d1bb930fc1eb" exitCode=0
Sep 16 15:50:07.029529 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.026079    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal" event={"ID":"80e7377d29e102f842e6625acdbbaa55","Type":"ContainerDied","Data":"fde308fa7489fc916edadaf5b74cd8f1671eee7adb7214b92dc8d1bb930fc1eb"}
Sep 16 15:50:07.031112 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.031028    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nt24x" event={"ID":"fe3174a1-8fb2-402d-9826-df2eb068dfcb","Type":"ContainerStarted","Data":"09528dc36f09bb8682c62c08f59cfc051346ea6806563218767e22ae3f28b0c5"}
Sep 16 15:50:07.041423 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.041396    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9" event={"ID":"1a7785e3-0b97-40ee-8fb8-b4c767aadf11","Type":"ContainerStarted","Data":"21a998e07cb6eb4c95475dbac8f9caee5ee66a7349e43aaeed43f11856ab44ec"}
Sep 16 15:50:07.053612 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.053503    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s2k52" event={"ID":"9a690936-8677-451d-8ff9-c6b3b92d666a","Type":"ContainerStarted","Data":"44dadb38a388c146e4955a8c6e9f789430769496847c17dde5198c7e8a99a77b"}
Sep 16 15:50:07.063004 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.062911    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w8vv" event={"ID":"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca","Type":"ContainerStarted","Data":"11f21693a598c405aeedf0ff2b4d525a06fa582ac27f2632be24bcaa1490ff8c"}
Sep 16 15:50:07.067726 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.067637    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nz7nc" event={"ID":"12c61b9c-b819-4330-bcf8-05341ce39ac8","Type":"ContainerStarted","Data":"9e8c378455d38be529963d29109c74007bb1481aa35d097b3f51af892c727dc2"}
Sep 16 15:50:07.069755 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.069680    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7jbf9" event={"ID":"c685b2f9-18ae-4c7f-b9c7-8f271c676488","Type":"ContainerStarted","Data":"7b360ef2df749536fc794e22403ccdfa3a43b7dda9b01d4bb609814117433536"}
Sep 16 15:50:07.531382 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.531283    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:07.531571 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:07.531457    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:07.531571 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:07.531549    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs podName:1332ae3f-698f-4d0e-9801-2aa87121a12b nodeName:}" failed. No retries permitted until 2026-09-16 15:50:09.531511983 +0000 UTC m=+6.051134756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs") pod "network-metrics-daemon-bbfmm" (UID: "1332ae3f-698f-4d0e-9801-2aa87121a12b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:07.632513 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.632466    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:07.632804 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:07.632638    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:07.632804 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:07.632709    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret podName:9823bb5e-f7d2-442d-b83b-c833dafb42ba nodeName:}" failed. No retries permitted until 2026-09-16 15:50:09.632687946 +0000 UTC m=+6.152310706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret") pod "global-pull-secret-syncer-86l22" (UID: "9823bb5e-f7d2-442d-b83b-c833dafb42ba") : object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:07.733517 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.733475    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7sf2\" (UniqueName: \"kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2\") pod \"network-check-target-7knnk\" (UID: \"76cadf23-c5f4-4d75-98ab-48509fd8cb42\") " pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:07.733794 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:07.733772    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 15:50:07.733878 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:07.733801    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 15:50:07.733878 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:07.733815    2574 projected.go:196] Error preparing data for projected volume kube-api-access-z7sf2 for pod openshift-network-diagnostics/network-check-target-7knnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:07.733991 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:07.733880    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2 podName:76cadf23-c5f4-4d75-98ab-48509fd8cb42 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:09.733860525 +0000 UTC m=+6.253483298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7sf2" (UniqueName: "kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2") pod "network-check-target-7knnk" (UID: "76cadf23-c5f4-4d75-98ab-48509fd8cb42") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:07.979829 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.979757    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 15:45:06 +0000 UTC" deadline="2028-06-03 21:44:51.346555628 +0000 UTC"
Sep 16 15:50:07.979829 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:07.979826    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15029h54m43.366733763s"
Sep 16 15:50:08.002151 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:08.002087    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:08.002151 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:08.002106    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:08.002393 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:08.002221    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:08.002783 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:08.002730    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:08.084374 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:08.083454    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal" event={"ID":"80e7377d29e102f842e6625acdbbaa55","Type":"ContainerStarted","Data":"bd48d1f6c7d5be988a464ea0d8c4b20e8a1cb389bbd416dd0982eee1b0f952d8"}
Sep 16 15:50:08.097791 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:08.097719    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-143.ec2.internal" podStartSLOduration=3.097697041 podStartE2EDuration="3.097697041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:08.097450354 +0000 UTC m=+4.617073137" watchObservedRunningTime="2026-09-16 15:50:08.097697041 +0000 UTC m=+4.617319823"
Sep 16 15:50:08.988999 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:08.988966    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:08.989714 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:08.989098    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:09.549130 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:09.549090    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:09.549363 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:09.549331    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:09.549438 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:09.549406    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs podName:1332ae3f-698f-4d0e-9801-2aa87121a12b nodeName:}" failed. No retries permitted until 2026-09-16 15:50:13.549386621 +0000 UTC m=+10.069009393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs") pod "network-metrics-daemon-bbfmm" (UID: "1332ae3f-698f-4d0e-9801-2aa87121a12b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:09.649619 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:09.649561    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:09.649847 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:09.649732    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:09.649847 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:09.649800    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret podName:9823bb5e-f7d2-442d-b83b-c833dafb42ba nodeName:}" failed. No retries permitted until 2026-09-16 15:50:13.649780746 +0000 UTC m=+10.169403518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret") pod "global-pull-secret-syncer-86l22" (UID: "9823bb5e-f7d2-442d-b83b-c833dafb42ba") : object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:09.750963 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:09.750921    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7sf2\" (UniqueName: \"kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2\") pod \"network-check-target-7knnk\" (UID: \"76cadf23-c5f4-4d75-98ab-48509fd8cb42\") " pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:09.751165 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:09.751134    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 15:50:09.751165 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:09.751157    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 15:50:09.751293 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:09.751171    2574 projected.go:196] Error preparing data for projected volume kube-api-access-z7sf2 for pod openshift-network-diagnostics/network-check-target-7knnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:09.751293 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:09.751234    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2 podName:76cadf23-c5f4-4d75-98ab-48509fd8cb42 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:13.75121473 +0000 UTC m=+10.270837490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7sf2" (UniqueName: "kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2") pod "network-check-target-7knnk" (UID: "76cadf23-c5f4-4d75-98ab-48509fd8cb42") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:09.993475 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:09.993418    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:09.993989 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:09.993549    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:09.993989 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:09.993590    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:09.993989 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:09.993722    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:10.989148 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:10.989107    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:10.989320 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:10.989248    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:11.989185 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:11.989145    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:11.989668 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:11.989292    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:11.989668 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:11.989460    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:11.989668 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:11.989561    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:12.988909 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:12.988672    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:12.988909 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:12.988806    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:13.585934 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:13.585896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:13.586569 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:13.586082    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:13.586569 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:13.586160    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs podName:1332ae3f-698f-4d0e-9801-2aa87121a12b nodeName:}" failed. No retries permitted until 2026-09-16 15:50:21.586139326 +0000 UTC m=+18.105762107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs") pod "network-metrics-daemon-bbfmm" (UID: "1332ae3f-698f-4d0e-9801-2aa87121a12b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:13.687300 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:13.687258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:13.687475 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:13.687411    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:13.687580 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:13.687480    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret podName:9823bb5e-f7d2-442d-b83b-c833dafb42ba nodeName:}" failed. No retries permitted until 2026-09-16 15:50:21.687460435 +0000 UTC m=+18.207083190 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret") pod "global-pull-secret-syncer-86l22" (UID: "9823bb5e-f7d2-442d-b83b-c833dafb42ba") : object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:13.789000 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:13.788387    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7sf2\" (UniqueName: \"kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2\") pod \"network-check-target-7knnk\" (UID: \"76cadf23-c5f4-4d75-98ab-48509fd8cb42\") " pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:13.789000 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:13.788609    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 15:50:13.789000 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:13.788633    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 15:50:13.789000 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:13.788644    2574 projected.go:196] Error preparing data for projected volume kube-api-access-z7sf2 for pod openshift-network-diagnostics/network-check-target-7knnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:13.789000 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:13.788688    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2 podName:76cadf23-c5f4-4d75-98ab-48509fd8cb42 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:21.788674691 +0000 UTC m=+18.308297446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7sf2" (UniqueName: "kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2") pod "network-check-target-7knnk" (UID: "76cadf23-c5f4-4d75-98ab-48509fd8cb42") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:13.992601 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:13.992530    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:13.992820 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:13.992640    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:13.992820 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:13.992689    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:13.992820 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:13.992803    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:14.988220 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:14.988188    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:14.988777 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:14.988315    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:15.991095 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:15.991062    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:15.991095 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:15.991081    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:15.991605 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:15.991174    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:15.991605 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:15.991292    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:16.988487 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:16.988450    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:16.988669 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:16.988615    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:17.991609 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:17.991588    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:17.991986 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:17.991588    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:17.991986 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:17.991747    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:17.991986 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:17.991842    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:18.988431 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:18.988400    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:18.988614 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:18.988535    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:19.989118 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:19.989083    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:19.989568 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:19.989092    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:19.989568 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:19.989217    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:19.989568 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:19.989322    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:20.988865 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:20.988824    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:20.989092 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:20.988956    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:21.644107 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:21.644070    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:21.644502 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:21.644222    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:21.644502 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:21.644290    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs podName:1332ae3f-698f-4d0e-9801-2aa87121a12b nodeName:}" failed. No retries permitted until 2026-09-16 15:50:37.644272566 +0000 UTC m=+34.163895321 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs") pod "network-metrics-daemon-bbfmm" (UID: "1332ae3f-698f-4d0e-9801-2aa87121a12b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:21.745197 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:21.745164    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:21.745367 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:21.745283    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:21.745367 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:21.745339    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret podName:9823bb5e-f7d2-442d-b83b-c833dafb42ba nodeName:}" failed. No retries permitted until 2026-09-16 15:50:37.745321529 +0000 UTC m=+34.264944285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret") pod "global-pull-secret-syncer-86l22" (UID: "9823bb5e-f7d2-442d-b83b-c833dafb42ba") : object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:21.845570 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:21.845532    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7sf2\" (UniqueName: \"kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2\") pod \"network-check-target-7knnk\" (UID: \"76cadf23-c5f4-4d75-98ab-48509fd8cb42\") " pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:21.845768 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:21.845722    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 15:50:21.845768 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:21.845749    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 15:50:21.845768 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:21.845761    2574 projected.go:196] Error preparing data for projected volume kube-api-access-z7sf2 for pod openshift-network-diagnostics/network-check-target-7knnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:21.845918 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:21.845830    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2 podName:76cadf23-c5f4-4d75-98ab-48509fd8cb42 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:37.845809798 +0000 UTC m=+34.365432556 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7sf2" (UniqueName: "kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2") pod "network-check-target-7knnk" (UID: "76cadf23-c5f4-4d75-98ab-48509fd8cb42") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:21.993012 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:21.992935    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:21.993012 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:21.992974    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:21.993220 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:21.993057    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:21.993270 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:21.993212    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:22.988431 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:22.988389    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:22.988915 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:22.988548    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:23.988877 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:23.988843    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:23.989280 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:23.988969    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:23.989280 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:23.989056    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:23.989280 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:23.989185    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:24.988914 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:24.988884    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:24.989259 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:24.988984    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:25.989593 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:25.989185    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:25.990356 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:25.989193    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:25.990356 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:25.989713    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:25.990356 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:25.989790    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:26.130974 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.130940    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9" event={"ID":"1a7785e3-0b97-40ee-8fb8-b4c767aadf11","Type":"ContainerStarted","Data":"0515085dab787a472fd258a085e285110bb0694e25d78a1ff1e1e433ef1576f2"}
Sep 16 15:50:26.132347 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.132322    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s2k52" event={"ID":"9a690936-8677-451d-8ff9-c6b3b92d666a","Type":"ContainerStarted","Data":"00c9ece418e9a321566fe1b93bcd6056115c044711b1a4e4d1488b58ea497b4f"}
Sep 16 15:50:26.133916 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.133893    2574 generic.go:356] "Generic (PLEG): container finished" podID="9fe8739d-6acf-41c1-b1c8-f11a8fa361ca" containerID="6cc0c50c8dccb22134372f00addf82961dc8146de0bdd782462f4ea36378c09d" exitCode=0
Sep 16 15:50:26.134000 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.133977    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w8vv" event={"ID":"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca","Type":"ContainerDied","Data":"6cc0c50c8dccb22134372f00addf82961dc8146de0bdd782462f4ea36378c09d"}
Sep 16 15:50:26.137823 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.137785    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nz7nc" event={"ID":"12c61b9c-b819-4330-bcf8-05341ce39ac8","Type":"ContainerStarted","Data":"5a53b151709fd13537f5207ea79fed8be71f2162f0a7869f948bc7ca956dc843"}
Sep 16 15:50:26.139425 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.139399    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7jbf9" event={"ID":"c685b2f9-18ae-4c7f-b9c7-8f271c676488","Type":"ContainerStarted","Data":"0619a06963e28be493b8b8048102f0a4825de614edf20a925a1d12b459367e60"}
Sep 16 15:50:26.140816 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.140797    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6wnxz" event={"ID":"46cd7e46-0254-4c9c-b274-136e18ba7d91","Type":"ContainerStarted","Data":"24f07cce5199f7ed6b6aa28027d1f2add9f667d46be8a7d2657abd893fcc4a3e"}
Sep 16 15:50:26.144241 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.144214    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94" event={"ID":"5b155b34-181a-43f2-a2e7-a4c540287735","Type":"ContainerStarted","Data":"78f1c6e6d9b4fda869afa8629b1ccebdb28caae14f7f001d8386831f268dbc3b"}
Sep 16 15:50:26.144352 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.144276    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94" event={"ID":"5b155b34-181a-43f2-a2e7-a4c540287735","Type":"ContainerStarted","Data":"2db44f0198175989699ef9511ca244cb7b5bd33d8cde517e935d87ef7d9ffd65"}
Sep 16 15:50:26.144352 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.144321    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94" event={"ID":"5b155b34-181a-43f2-a2e7-a4c540287735","Type":"ContainerStarted","Data":"d3b6ee8debdaa3113c612b20e6d1558011b3dbf4050403b83c152106ca16b6a2"}
Sep 16 15:50:26.144451 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.144362    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94" event={"ID":"5b155b34-181a-43f2-a2e7-a4c540287735","Type":"ContainerStarted","Data":"e843026f160c7d435eeedba3fca541df3f22fda99d3eb3f82f5f0134bba9bfaf"}
Sep 16 15:50:26.144451 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.144402    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94" event={"ID":"5b155b34-181a-43f2-a2e7-a4c540287735","Type":"ContainerStarted","Data":"97b747d26629d99df75e6a95436065e62afeca5dc6990689f5dc6767120ddda9"}
Sep 16 15:50:26.144547 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.144452    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94" event={"ID":"5b155b34-181a-43f2-a2e7-a4c540287735","Type":"ContainerStarted","Data":"d7f413d2302efbfa862e7939cfe4637a33628fe2fb6ca5da892d1e4ca1909b6c"}
Sep 16 15:50:26.145613 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.145596    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6r28d" event={"ID":"eedc6ccd-a007-4ca4-98ba-54fa62367075","Type":"ContainerStarted","Data":"06a56a16d453cbad82bdbb8dcc730955d9e0fd1afbf50a772734f08afab3ec51"}
Sep 16 15:50:26.146938 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.146918    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-143.ec2.internal" event={"ID":"ed97d765e2c9a80c45b1435ad72f4818","Type":"ContainerStarted","Data":"68da04869ff1c1be582d850e58ef87b627e635595e1e86349acebc783d22ab65"}
Sep 16 15:50:26.147732 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.147700    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s2k52" podStartSLOduration=3.46267134 podStartE2EDuration="22.14768857s" totalImagesPullingTime="18.68501723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:26.14764849 +0000 UTC m=+22.667271280" watchObservedRunningTime="2026-09-16 15:50:26.14768857 +0000 UTC m=+22.667311387"
Sep 16 15:50:26.163469 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.163433    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7jbf9" podStartSLOduration=3.41038274 podStartE2EDuration="22.16342239s" totalImagesPullingTime="18.75303965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:26.163116145 +0000 UTC m=+22.682738924" watchObservedRunningTime="2026-09-16 15:50:26.16342239 +0000 UTC m=+22.683045166"
Sep 16 15:50:26.177116 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.177081    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6r28d" podStartSLOduration=3.488667598 podStartE2EDuration="22.177070656s" totalImagesPullingTime="18.688403058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:26.17676369 +0000 UTC m=+22.696386470" watchObservedRunningTime="2026-09-16 15:50:26.177070656 +0000 UTC m=+22.696693432"
Sep 16 15:50:26.215370 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.215327    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6wnxz" podStartSLOduration=3.510835399 podStartE2EDuration="22.215314716s" totalImagesPullingTime="18.704479317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:26.214788977 +0000 UTC m=+22.734411753" watchObservedRunningTime="2026-09-16 15:50:26.215314716 +0000 UTC m=+22.734937492"
Sep 16 15:50:26.242255 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.242192    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nz7nc" podStartSLOduration=3.574647378 podStartE2EDuration="22.24218026s" totalImagesPullingTime="18.667532882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:26.241982664 +0000 UTC m=+22.761605443" watchObservedRunningTime="2026-09-16 15:50:26.24218026 +0000 UTC m=+22.761803036"
Sep 16 15:50:26.242453 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.242435    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-143.ec2.internal" podStartSLOduration=21.242430965 podStartE2EDuration="21.242430965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:26.22687986 +0000 UTC m=+22.746502636" watchObservedRunningTime="2026-09-16 15:50:26.242430965 +0000 UTC m=+22.762053761"
Sep 16 15:50:26.421456 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.421411    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 15:50:26.967645 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.967538    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T15:50:26.421429195Z","UUID":"8e8d600b-42c9-4eb7-b653-1bc505f004fa","Handler":null,"Name":"","Endpoint":""}
Sep 16 15:50:26.969195 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.969171    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 15:50:26.969329 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.969207    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 15:50:26.988419 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:26.988389    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:26.988613 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:26.988512    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:27.154349 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:27.154290    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nt24x" event={"ID":"fe3174a1-8fb2-402d-9826-df2eb068dfcb","Type":"ContainerStarted","Data":"911006039a0c15bb197556778f52d9780008066f7348f044881ece1ba65743da"}
Sep 16 15:50:27.156532 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:27.156489    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9" event={"ID":"1a7785e3-0b97-40ee-8fb8-b4c767aadf11","Type":"ContainerStarted","Data":"20d49a1eba5da9036bbd0864c4434226f4f9e7b4c2578230d0a6ea72ca59b215"}
Sep 16 15:50:27.819075 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:27.818842    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s2k52"
Sep 16 15:50:27.992529 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:27.992489    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:27.992777 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:27.992499    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:27.992777 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:27.992616    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:27.992777 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:27.992719    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:28.163435 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:28.163394    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94" event={"ID":"5b155b34-181a-43f2-a2e7-a4c540287735","Type":"ContainerStarted","Data":"3c4c0da9931d907d7b4e367ecdc6efa19e54466afd725eda3bc28509788dbddb"}
Sep 16 15:50:28.651500 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:28.651458    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s2k52"
Sep 16 15:50:28.652077 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:28.652056    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s2k52"
Sep 16 15:50:28.672974 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:28.672930    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nt24x" podStartSLOduration=6.005675359 podStartE2EDuration="24.672908017s" totalImagesPullingTime="18.667232658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:27.167958364 +0000 UTC m=+23.687581144" watchObservedRunningTime="2026-09-16 15:50:28.672908017 +0000 UTC m=+25.192530793"
Sep 16 15:50:28.988729 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:28.988647    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:28.988915 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:28.988793    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:29.166282 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:29.166255    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s2k52"
Sep 16 15:50:29.988592 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:29.988559    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:29.988779 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:29.988702    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:29.988779 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:29.988758    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:29.988889 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:29.988869    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:30.988161 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:30.988134    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:30.988566 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:30.988233    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:31.171909 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:31.171807    2574 generic.go:356] "Generic (PLEG): container finished" podID="9fe8739d-6acf-41c1-b1c8-f11a8fa361ca" containerID="464f9cdf4b22bae3c779c0f766e1971a4aa4db7a0e1c7629637a6446730b9e19" exitCode=0
Sep 16 15:50:31.171909 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:31.171893    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w8vv" event={"ID":"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca","Type":"ContainerDied","Data":"464f9cdf4b22bae3c779c0f766e1971a4aa4db7a0e1c7629637a6446730b9e19"}
Sep 16 15:50:31.175884 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:31.175857    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94" event={"ID":"5b155b34-181a-43f2-a2e7-a4c540287735","Type":"ContainerStarted","Data":"578c166b9f276bba1879717ffd6051ecbb0c0c28dcde62ed10ddf519782ca55b"}
Sep 16 15:50:31.176137 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:31.176119    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:31.176229 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:31.176183    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:31.176292 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:31.176234    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:31.178991 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:31.178957    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9" event={"ID":"1a7785e3-0b97-40ee-8fb8-b4c767aadf11","Type":"ContainerStarted","Data":"e94289f6b86d708c8c7287fab7821752e353e1658c0dfe4f580351cf8a08b25d"}
Sep 16 15:50:31.191492 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:31.191460    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:31.191636 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:31.191587    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:50:31.208502 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:31.208456    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bc8r9" podStartSLOduration=2.9637501630000003 podStartE2EDuration="27.208440515s" totalImagesPullingTime="24.244690352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:31.208174183 +0000 UTC m=+27.727796960" watchObservedRunningTime="2026-09-16 15:50:31.208440515 +0000 UTC m=+27.728063291"
Sep 16 15:50:31.235644 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:31.235588    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94" podStartSLOduration=8.478697223 podStartE2EDuration="27.235572086s" totalImagesPullingTime="18.756874863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:31.235221834 +0000 UTC m=+27.754844609" watchObservedRunningTime="2026-09-16 15:50:31.235572086 +0000 UTC m=+27.755194862"
Sep 16 15:50:31.991909 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:31.991882    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:31.992504 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:31.991884    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:31.992504 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:31.991998    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:31.992504 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:31.992071    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:32.101975 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:32.101939    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-86l22"]
Sep 16 15:50:32.102126 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:32.102048    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:32.102190 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:32.102164    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:32.102728 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:32.102700    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7knnk"]
Sep 16 15:50:32.104865 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:32.104839    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bbfmm"]
Sep 16 15:50:32.181073 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:32.180995    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:32.181073 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:32.181005    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:32.181258 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:32.181229    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:32.181893 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:32.181707    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:33.185202 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:33.185011    2574 generic.go:356] "Generic (PLEG): container finished" podID="9fe8739d-6acf-41c1-b1c8-f11a8fa361ca" containerID="29ec7032df487a7cf8ff56a2f88247a1548f42f2ddc9ae3c457da70ef9266dc9" exitCode=0
Sep 16 15:50:33.185710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:33.185098    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w8vv" event={"ID":"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca","Type":"ContainerDied","Data":"29ec7032df487a7cf8ff56a2f88247a1548f42f2ddc9ae3c457da70ef9266dc9"}
Sep 16 15:50:33.993232 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:33.993202    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:33.993559 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:33.993213    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:33.993649 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:33.993232    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:33.993714 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:33.993687    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:33.993817 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:33.993770    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:33.993934 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:33.993810    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:35.191656 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:35.191576    2574 generic.go:356] "Generic (PLEG): container finished" podID="9fe8739d-6acf-41c1-b1c8-f11a8fa361ca" containerID="94f25f250a4191da0bf54dba7f07230aa6ff6993c93437df775e8fec2458c4ca" exitCode=0
Sep 16 15:50:35.191656 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:35.191638    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w8vv" event={"ID":"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca","Type":"ContainerDied","Data":"94f25f250a4191da0bf54dba7f07230aa6ff6993c93437df775e8fec2458c4ca"}
Sep 16 15:50:35.988409 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:35.988369    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:35.988618 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:35.988369    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:35.988618 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:35.988505    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-86l22" podUID="9823bb5e-f7d2-442d-b83b-c833dafb42ba"
Sep 16 15:50:35.988618 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:35.988369    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:35.988780 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:35.988624    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:50:35.988780 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:35.988699    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knnk" podUID="76cadf23-c5f4-4d75-98ab-48509fd8cb42"
Sep 16 15:50:37.651583 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.651546    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:37.651948 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:37.651693    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:37.651948 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:37.651760    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs podName:1332ae3f-698f-4d0e-9801-2aa87121a12b nodeName:}" failed. No retries permitted until 2026-09-16 15:51:09.651739976 +0000 UTC m=+66.171362734 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs") pod "network-metrics-daemon-bbfmm" (UID: "1332ae3f-698f-4d0e-9801-2aa87121a12b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 15:50:37.752485 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.752447    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:37.752663 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:37.752621    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:37.752708 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:37.752691    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret podName:9823bb5e-f7d2-442d-b83b-c833dafb42ba nodeName:}" failed. No retries permitted until 2026-09-16 15:51:09.752674679 +0000 UTC m=+66.272297438 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret") pod "global-pull-secret-syncer-86l22" (UID: "9823bb5e-f7d2-442d-b83b-c833dafb42ba") : object "kube-system"/"original-pull-secret" not registered
Sep 16 15:50:37.789732 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.789701    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-143.ec2.internal" event="NodeReady"
Sep 16 15:50:37.789885 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.789851    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 15:50:37.847324 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.847293    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wq8bz"]
Sep 16 15:50:37.852263 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.852238    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-twc8n"]
Sep 16 15:50:37.852434 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.852413    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:37.852897 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.852790    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7sf2\" (UniqueName: \"kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2\") pod \"network-check-target-7knnk\" (UID: \"76cadf23-c5f4-4d75-98ab-48509fd8cb42\") " pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:37.853019 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:37.852954    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 15:50:37.853019 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:37.852976    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 15:50:37.853019 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:37.852988    2574 projected.go:196] Error preparing data for projected volume kube-api-access-z7sf2 for pod openshift-network-diagnostics/network-check-target-7knnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:37.853176 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:37.853044    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2 podName:76cadf23-c5f4-4d75-98ab-48509fd8cb42 nodeName:}" failed. No retries permitted until 2026-09-16 15:51:09.853028732 +0000 UTC m=+66.372651488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7sf2" (UniqueName: "kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2") pod "network-check-target-7knnk" (UID: "76cadf23-c5f4-4d75-98ab-48509fd8cb42") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 15:50:37.854694 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.854672    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 15:50:37.854786 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.854682    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 15:50:37.854786 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.854765    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-thq8g\""
Sep 16 15:50:37.856249 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.856228    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-twc8n"
Sep 16 15:50:37.859321 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.859302    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j6n8k\""
Sep 16 15:50:37.859428 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.859351    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 15:50:37.859576 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.859532    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 15:50:37.859576 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.859564    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 15:50:37.877425 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.877350    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-twc8n"]
Sep 16 15:50:37.878356 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.878331    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wq8bz"]
Sep 16 15:50:37.953917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.953880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb25687b-a293-40ae-a3a7-e2357384bcd4-tmp-dir\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:37.954111 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.953927    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n47qr\" (UniqueName: \"kubernetes.io/projected/cb25687b-a293-40ae-a3a7-e2357384bcd4-kube-api-access-n47qr\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:37.954111 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.953968    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsckt\" (UniqueName: \"kubernetes.io/projected/1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59-kube-api-access-xsckt\") pod \"ingress-canary-twc8n\" (UID: \"1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59\") " pod="openshift-ingress-canary/ingress-canary-twc8n"
Sep 16 15:50:37.954111 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.954006    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb25687b-a293-40ae-a3a7-e2357384bcd4-config-volume\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:37.954111 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.954031    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:37.954289 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.954104    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59-cert\") pod \"ingress-canary-twc8n\" (UID: \"1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59\") " pod="openshift-ingress-canary/ingress-canary-twc8n"
Sep 16 15:50:37.989151 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.989114    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:50:37.989151 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.989141    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:50:37.989377 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.989120    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:50:37.992307 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.991914    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hlg9r\""
Sep 16 15:50:37.992307 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.991924    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 15:50:37.992307 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.992007    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 15:50:37.992307 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.992011    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 15:50:37.992307 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.992108    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mmzn4\""
Sep 16 15:50:37.992307 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:37.992156    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 15:50:38.054498 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:38.054457    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsckt\" (UniqueName: \"kubernetes.io/projected/1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59-kube-api-access-xsckt\") pod \"ingress-canary-twc8n\" (UID: \"1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59\") " pod="openshift-ingress-canary/ingress-canary-twc8n"
Sep 16 15:50:38.054651 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:38.054509    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb25687b-a293-40ae-a3a7-e2357384bcd4-config-volume\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:38.054651 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:38.054574    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:38.054651 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:38.054603    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59-cert\") pod \"ingress-canary-twc8n\" (UID: \"1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59\") " pod="openshift-ingress-canary/ingress-canary-twc8n"
Sep 16 15:50:38.054805 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:38.054672    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb25687b-a293-40ae-a3a7-e2357384bcd4-tmp-dir\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:38.054805 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:38.054698    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47qr\" (UniqueName: \"kubernetes.io/projected/cb25687b-a293-40ae-a3a7-e2357384bcd4-kube-api-access-n47qr\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:38.054911 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:38.054883    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:50:38.055009 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:38.054954    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls podName:cb25687b-a293-40ae-a3a7-e2357384bcd4 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:38.554937245 +0000 UTC m=+35.074560012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls") pod "dns-default-wq8bz" (UID: "cb25687b-a293-40ae-a3a7-e2357384bcd4") : secret "dns-default-metrics-tls" not found
Sep 16 15:50:38.055500 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:38.055466    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb25687b-a293-40ae-a3a7-e2357384bcd4-tmp-dir\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:38.055639 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:38.055618    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb25687b-a293-40ae-a3a7-e2357384bcd4-config-volume\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:38.060483 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:38.060461    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59-cert\") pod \"ingress-canary-twc8n\" (UID: \"1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59\") " pod="openshift-ingress-canary/ingress-canary-twc8n"
Sep 16 15:50:38.063469 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:38.063444    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47qr\" (UniqueName: \"kubernetes.io/projected/cb25687b-a293-40ae-a3a7-e2357384bcd4-kube-api-access-n47qr\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:38.063591 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:38.063452    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsckt\" (UniqueName: \"kubernetes.io/projected/1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59-kube-api-access-xsckt\") pod \"ingress-canary-twc8n\" (UID: \"1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59\") " pod="openshift-ingress-canary/ingress-canary-twc8n"
Sep 16 15:50:38.169251 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:38.169170    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-twc8n"
Sep 16 15:50:38.558566 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:38.558466    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:38.558718 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:38.558635    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:50:38.558718 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:38.558695    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls podName:cb25687b-a293-40ae-a3a7-e2357384bcd4 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:39.55867894 +0000 UTC m=+36.078301707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls") pod "dns-default-wq8bz" (UID: "cb25687b-a293-40ae-a3a7-e2357384bcd4") : secret "dns-default-metrics-tls" not found
Sep 16 15:50:39.205965 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:39.205937    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w8vv" event={"ID":"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca","Type":"ContainerStarted","Data":"ac5f59ceef3189c3aca3563f6b94073ab042b40dbf7896f95158df02645e291d"}
Sep 16 15:50:39.312542 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:39.312459    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-twc8n"]
Sep 16 15:50:39.333465 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:50:39.333431    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1a3f5c_a7cc_45cd_8ff2_e04f658cdb59.slice/crio-02aa27ca68825139cceafb03c65fdcb1d61b8f5217db79c1b2a8f4da5daf1690 WatchSource:0}: Error finding container 02aa27ca68825139cceafb03c65fdcb1d61b8f5217db79c1b2a8f4da5daf1690: Status 404 returned error can't find the container with id 02aa27ca68825139cceafb03c65fdcb1d61b8f5217db79c1b2a8f4da5daf1690
Sep 16 15:50:39.565720 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:39.565640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:39.565853 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:39.565774    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:50:39.565853 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:39.565837    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls podName:cb25687b-a293-40ae-a3a7-e2357384bcd4 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:41.565824173 +0000 UTC m=+38.085446928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls") pod "dns-default-wq8bz" (UID: "cb25687b-a293-40ae-a3a7-e2357384bcd4") : secret "dns-default-metrics-tls" not found
Sep 16 15:50:40.210764 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:40.210728    2574 generic.go:356] "Generic (PLEG): container finished" podID="9fe8739d-6acf-41c1-b1c8-f11a8fa361ca" containerID="ac5f59ceef3189c3aca3563f6b94073ab042b40dbf7896f95158df02645e291d" exitCode=0
Sep 16 15:50:40.211293 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:40.210807    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w8vv" event={"ID":"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca","Type":"ContainerDied","Data":"ac5f59ceef3189c3aca3563f6b94073ab042b40dbf7896f95158df02645e291d"}
Sep 16 15:50:40.212078 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:40.212059    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-twc8n" event={"ID":"1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59","Type":"ContainerStarted","Data":"02aa27ca68825139cceafb03c65fdcb1d61b8f5217db79c1b2a8f4da5daf1690"}
Sep 16 15:50:41.217294 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:41.217260    2574 generic.go:356] "Generic (PLEG): container finished" podID="9fe8739d-6acf-41c1-b1c8-f11a8fa361ca" containerID="5ffc300619126095ac822fecd22e4da5665a03b2a3a4cb33c3cac9bbcd370ded" exitCode=0
Sep 16 15:50:41.217893 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:41.217322    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w8vv" event={"ID":"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca","Type":"ContainerDied","Data":"5ffc300619126095ac822fecd22e4da5665a03b2a3a4cb33c3cac9bbcd370ded"}
Sep 16 15:50:41.582187 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:41.582103    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:41.582360 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:41.582237    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:50:41.582360 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:41.582307    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls podName:cb25687b-a293-40ae-a3a7-e2357384bcd4 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:45.582287875 +0000 UTC m=+42.101910631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls") pod "dns-default-wq8bz" (UID: "cb25687b-a293-40ae-a3a7-e2357384bcd4") : secret "dns-default-metrics-tls" not found
Sep 16 15:50:42.223067 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:42.222870    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2w8vv" event={"ID":"9fe8739d-6acf-41c1-b1c8-f11a8fa361ca","Type":"ContainerStarted","Data":"cadda0b4d78d96b2ce423ecebdb5bb94dc536245b706fbd87764a73036da43b0"}
Sep 16 15:50:42.243866 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:42.243808    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2w8vv" podStartSLOduration=9.17519475 podStartE2EDuration="38.24379068s" totalImagesPullingTime="29.06859593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:42.242810453 +0000 UTC m=+38.762433230" watchObservedRunningTime="2026-09-16 15:50:42.24379068 +0000 UTC m=+38.763413457"
Sep 16 15:50:44.228625 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:44.228589    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-twc8n" event={"ID":"1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59","Type":"ContainerStarted","Data":"24eadc25dfbe6a48941ecb2fed8a41e193bfe964517b44c5672b70f22ee8d494"}
Sep 16 15:50:44.244196 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:44.244142    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-twc8n" podStartSLOduration=3.182797661 podStartE2EDuration="7.244126338s" totalImagesPullingTime="4.061328677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:50:44.243781644 +0000 UTC m=+40.763404421" watchObservedRunningTime="2026-09-16 15:50:44.244126338 +0000 UTC m=+40.763749114"
Sep 16 15:50:45.609774 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:45.609724    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:45.610150 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:45.609889    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:50:45.610150 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:45.609957    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls podName:cb25687b-a293-40ae-a3a7-e2357384bcd4 nodeName:}" failed. No retries permitted until 2026-09-16 15:50:53.609939687 +0000 UTC m=+50.129562456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls") pod "dns-default-wq8bz" (UID: "cb25687b-a293-40ae-a3a7-e2357384bcd4") : secret "dns-default-metrics-tls" not found
Sep 16 15:50:53.666715 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:50:53.666673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:50:53.667153 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:53.666788    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:50:53.667153 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:50:53.666852    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls podName:cb25687b-a293-40ae-a3a7-e2357384bcd4 nodeName:}" failed. No retries permitted until 2026-09-16 15:51:09.666837382 +0000 UTC m=+66.186460137 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls") pod "dns-default-wq8bz" (UID: "cb25687b-a293-40ae-a3a7-e2357384bcd4") : secret "dns-default-metrics-tls" not found
Sep 16 15:51:03.201798 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:03.201769    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldr94"
Sep 16 15:51:09.668943 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:09.668903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:51:09.669318 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:09.668963    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:51:09.669318 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:51:09.669055    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:51:09.669318 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:51:09.669132    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls podName:cb25687b-a293-40ae-a3a7-e2357384bcd4 nodeName:}" failed. No retries permitted until 2026-09-16 15:51:41.669111917 +0000 UTC m=+98.188734687 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls") pod "dns-default-wq8bz" (UID: "cb25687b-a293-40ae-a3a7-e2357384bcd4") : secret "dns-default-metrics-tls" not found
Sep 16 15:51:09.671501 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:09.671477    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 15:51:09.679642 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:51:09.679621    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 15:51:09.679705 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:51:09.679695    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs podName:1332ae3f-698f-4d0e-9801-2aa87121a12b nodeName:}" failed. No retries permitted until 2026-09-16 15:52:13.679679195 +0000 UTC m=+130.199301950 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs") pod "network-metrics-daemon-bbfmm" (UID: "1332ae3f-698f-4d0e-9801-2aa87121a12b") : secret "metrics-daemon-secret" not found
Sep 16 15:51:09.769282 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:09.769247    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:51:09.771839 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:09.771818    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 15:51:09.782141 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:09.782109    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9823bb5e-f7d2-442d-b83b-c833dafb42ba-original-pull-secret\") pod \"global-pull-secret-syncer-86l22\" (UID: \"9823bb5e-f7d2-442d-b83b-c833dafb42ba\") " pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:51:09.801351 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:09.801306    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-86l22"
Sep 16 15:51:09.870120 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:09.870087    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7sf2\" (UniqueName: \"kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2\") pod \"network-check-target-7knnk\" (UID: \"76cadf23-c5f4-4d75-98ab-48509fd8cb42\") " pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:51:09.872908 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:09.872888    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 15:51:09.882431 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:09.882409    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 15:51:09.893299 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:09.893271    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7sf2\" (UniqueName: \"kubernetes.io/projected/76cadf23-c5f4-4d75-98ab-48509fd8cb42-kube-api-access-z7sf2\") pod \"network-check-target-7knnk\" (UID: \"76cadf23-c5f4-4d75-98ab-48509fd8cb42\") " pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:51:10.068282 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:10.068195    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-86l22"]
Sep 16 15:51:10.071312 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:51:10.071272    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9823bb5e_f7d2_442d_b83b_c833dafb42ba.slice/crio-2e7dd61d9712cb6150eb0af8f1268ebed3878f61419cb380cf2b68e601c24951 WatchSource:0}: Error finding container 2e7dd61d9712cb6150eb0af8f1268ebed3878f61419cb380cf2b68e601c24951: Status 404 returned error can't find the container with id 2e7dd61d9712cb6150eb0af8f1268ebed3878f61419cb380cf2b68e601c24951
Sep 16 15:51:10.109672 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:10.109645    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hlg9r\""
Sep 16 15:51:10.118470 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:10.118440    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:51:10.287768 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:10.287732    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-86l22" event={"ID":"9823bb5e-f7d2-442d-b83b-c833dafb42ba","Type":"ContainerStarted","Data":"2e7dd61d9712cb6150eb0af8f1268ebed3878f61419cb380cf2b68e601c24951"}
Sep 16 15:51:10.386082 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:10.386044    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7knnk"]
Sep 16 15:51:10.389858 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:51:10.389833    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cadf23_c5f4_4d75_98ab_48509fd8cb42.slice/crio-768c6edff0d0753b1984b4d21d7c12f04785be711da9092b1d06089f612da917 WatchSource:0}: Error finding container 768c6edff0d0753b1984b4d21d7c12f04785be711da9092b1d06089f612da917: Status 404 returned error can't find the container with id 768c6edff0d0753b1984b4d21d7c12f04785be711da9092b1d06089f612da917
Sep 16 15:51:11.292231 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:11.292158    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7knnk" event={"ID":"76cadf23-c5f4-4d75-98ab-48509fd8cb42","Type":"ContainerStarted","Data":"768c6edff0d0753b1984b4d21d7c12f04785be711da9092b1d06089f612da917"}
Sep 16 15:51:15.304046 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:15.304011    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-86l22" event={"ID":"9823bb5e-f7d2-442d-b83b-c833dafb42ba","Type":"ContainerStarted","Data":"23c112c59df3abd54f86fb5e847a59941f25c1c7295910381e42034b53564b6f"}
Sep 16 15:51:15.305469 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:15.305440    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7knnk" event={"ID":"76cadf23-c5f4-4d75-98ab-48509fd8cb42","Type":"ContainerStarted","Data":"07cce5ecd8a6d92aec2e3d374c213866d90dff691cb305e2ef06f853e93eb4df"}
Sep 16 15:51:15.305605 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:15.305576    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:51:15.331347 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:15.331280    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-86l22" podStartSLOduration=66.647620975 podStartE2EDuration="1m11.331260795s" totalImagesPullingTime="4.68363982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:51:15.317913138 +0000 UTC m=+71.837535915" watchObservedRunningTime="2026-09-16 15:51:15.331260795 +0000 UTC m=+71.850883570"
Sep 16 15:51:15.331640 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:15.331598    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7knnk" podStartSLOduration=66.96601962 podStartE2EDuration="1m11.331587699s" totalImagesPullingTime="4.365568079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:51:15.330853251 +0000 UTC m=+71.850476040" watchObservedRunningTime="2026-09-16 15:51:15.331587699 +0000 UTC m=+71.851210477"
Sep 16 15:51:41.683605 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:41.683572    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:51:41.683969 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:51:41.683728    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 15:51:41.683969 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:51:41.683803    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls podName:cb25687b-a293-40ae-a3a7-e2357384bcd4 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:45.683786846 +0000 UTC m=+162.203409601 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls") pod "dns-default-wq8bz" (UID: "cb25687b-a293-40ae-a3a7-e2357384bcd4") : secret "dns-default-metrics-tls" not found
Sep 16 15:51:46.312446 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:51:46.312412    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7knnk"
Sep 16 15:52:05.882922 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.882886    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-whb7g"]
Sep 16 15:52:05.885857 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.885833    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:05.888312 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.888287    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 15:52:05.888540 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.888508    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 15:52:05.888977 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.888959    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-t6x22\""
Sep 16 15:52:05.889164 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.889149    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 15:52:05.889404 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.889389    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 15:52:05.895029 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.895006    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 15:52:05.898701 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.898683    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-whb7g"]
Sep 16 15:52:05.946728 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.946695    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29cf843e-24ce-4d87-baea-993e52f3d2e4-trusted-ca-bundle\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:05.946870 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.946748    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp525\" (UniqueName: \"kubernetes.io/projected/29cf843e-24ce-4d87-baea-993e52f3d2e4-kube-api-access-qp525\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:05.946870 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.946804    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29cf843e-24ce-4d87-baea-993e52f3d2e4-serving-cert\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:05.946870 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.946864    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/29cf843e-24ce-4d87-baea-993e52f3d2e4-snapshots\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:05.946990 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.946884    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29cf843e-24ce-4d87-baea-993e52f3d2e4-service-ca-bundle\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:05.946990 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.946934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29cf843e-24ce-4d87-baea-993e52f3d2e4-tmp\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:05.986204 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.986172    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"]
Sep 16 15:52:05.989355 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.989332    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:05.994659 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.994631    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 15:52:05.994785 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.994700    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 15:52:05.994849 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.994697    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 15:52:05.994849 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.994811    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 15:52:05.994988 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.994924    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-br56r\""
Sep 16 15:52:05.995988 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.995968    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-79955bb76b-kbcfl"]
Sep 16 15:52:05.998740 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:05.998683    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.001677 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.001657    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 15:52:06.001787 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.001735    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 15:52:06.001871 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.001857    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 15:52:06.003223 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.003210    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 15:52:06.003447 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.003434    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 15:52:06.003487 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.003469    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 15:52:06.003832 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.003812    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kfwdc\""
Sep 16 15:52:06.005137 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.005110    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"]
Sep 16 15:52:06.015413 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.015391    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79955bb76b-kbcfl"]
Sep 16 15:52:06.047816 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.047785    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp525\" (UniqueName: \"kubernetes.io/projected/29cf843e-24ce-4d87-baea-993e52f3d2e4-kube-api-access-qp525\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:06.047816 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.047818    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-default-certificate\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.048005 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.047836    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-stats-auth\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.048005 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.047854    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29cf843e-24ce-4d87-baea-993e52f3d2e4-serving-cert\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:06.048005 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.047869    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kmqr\" (UniqueName: \"kubernetes.io/projected/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-kube-api-access-9kmqr\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.048005 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.047887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndlg5\" (UniqueName: \"kubernetes.io/projected/591b0e66-93df-447a-8ddd-d850e006a2a5-kube-api-access-ndlg5\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:06.048005 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.047931    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/591b0e66-93df-447a-8ddd-d850e006a2a5-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:06.048005 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.047967    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/29cf843e-24ce-4d87-baea-993e52f3d2e4-snapshots\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:06.048005 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.047989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29cf843e-24ce-4d87-baea-993e52f3d2e4-service-ca-bundle\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:06.048313 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.048039    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.048313 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.048068    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29cf843e-24ce-4d87-baea-993e52f3d2e4-tmp\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:06.048313 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.048085    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:06.048313 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.048114    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29cf843e-24ce-4d87-baea-993e52f3d2e4-trusted-ca-bundle\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:06.048313 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.048168    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.048581 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.048422    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29cf843e-24ce-4d87-baea-993e52f3d2e4-tmp\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:06.048749 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.048722    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29cf843e-24ce-4d87-baea-993e52f3d2e4-service-ca-bundle\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:06.048814 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.048746    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/29cf843e-24ce-4d87-baea-993e52f3d2e4-snapshots\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:06.049079 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.049056    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29cf843e-24ce-4d87-baea-993e52f3d2e4-trusted-ca-bundle\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:06.050217 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.050198    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29cf843e-24ce-4d87-baea-993e52f3d2e4-serving-cert\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:06.062453 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.062428    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp525\" (UniqueName: \"kubernetes.io/projected/29cf843e-24ce-4d87-baea-993e52f3d2e4-kube-api-access-qp525\") pod \"insights-operator-5844b856db-whb7g\" (UID: \"29cf843e-24ce-4d87-baea-993e52f3d2e4\") " pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:06.093962 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.093928    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44"]
Sep 16 15:52:06.097839 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.097823    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44"
Sep 16 15:52:06.100253 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.100230    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 15:52:06.100371 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.100230    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 15:52:06.100371 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.100232    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 15:52:06.100371 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.100314    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m8mk9\""
Sep 16 15:52:06.100371 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.100232    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 15:52:06.107242 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.107223    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44"]
Sep 16 15:52:06.148771 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.148711    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:06.148771 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.148747    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d89c5398-15a5-4797-9759-63216417b9a4-serving-cert\") pod \"service-ca-operator-cf7c776b7-65p44\" (UID: \"d89c5398-15a5-4797-9759-63216417b9a4\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44"
Sep 16 15:52:06.148928 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.148778    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.148928 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.148797    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-default-certificate\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.148928 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.148819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-stats-auth\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.148928 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:06.148843    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 15:52:06.148928 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:06.148862    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 15:52:06.148928 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.148848    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kmqr\" (UniqueName: \"kubernetes.io/projected/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-kube-api-access-9kmqr\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.148928 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:06.148908    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs podName:3b6415b2-6e76-454e-b85e-4e7e80cbd01e nodeName:}" failed. No retries permitted until 2026-09-16 15:52:06.64888961 +0000 UTC m=+123.168512383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs") pod "router-default-79955bb76b-kbcfl" (UID: "3b6415b2-6e76-454e-b85e-4e7e80cbd01e") : secret "router-metrics-certs-default" not found
Sep 16 15:52:06.149276 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:06.148995    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls podName:591b0e66-93df-447a-8ddd-d850e006a2a5 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:06.648975768 +0000 UTC m=+123.168598540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-28r4q" (UID: "591b0e66-93df-447a-8ddd-d850e006a2a5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 15:52:06.149276 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.149016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndlg5\" (UniqueName: \"kubernetes.io/projected/591b0e66-93df-447a-8ddd-d850e006a2a5-kube-api-access-ndlg5\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:06.149276 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.149046    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktjg8\" (UniqueName: \"kubernetes.io/projected/d89c5398-15a5-4797-9759-63216417b9a4-kube-api-access-ktjg8\") pod \"service-ca-operator-cf7c776b7-65p44\" (UID: \"d89c5398-15a5-4797-9759-63216417b9a4\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44"
Sep 16 15:52:06.149276 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.149074    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/591b0e66-93df-447a-8ddd-d850e006a2a5-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:06.149276 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.149105    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.149276 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.149125    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d89c5398-15a5-4797-9759-63216417b9a4-config\") pod \"service-ca-operator-cf7c776b7-65p44\" (UID: \"d89c5398-15a5-4797-9759-63216417b9a4\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44"
Sep 16 15:52:06.149594 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:06.149313    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle podName:3b6415b2-6e76-454e-b85e-4e7e80cbd01e nodeName:}" failed. No retries permitted until 2026-09-16 15:52:06.649297555 +0000 UTC m=+123.168920328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle") pod "router-default-79955bb76b-kbcfl" (UID: "3b6415b2-6e76-454e-b85e-4e7e80cbd01e") : configmap references non-existent config key: service-ca.crt
Sep 16 15:52:06.149732 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.149711    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/591b0e66-93df-447a-8ddd-d850e006a2a5-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:06.151276 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.151248    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-stats-auth\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.151367 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.151317    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-default-certificate\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.156870 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.156846    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kmqr\" (UniqueName: \"kubernetes.io/projected/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-kube-api-access-9kmqr\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.157268 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.157249    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndlg5\" (UniqueName: \"kubernetes.io/projected/591b0e66-93df-447a-8ddd-d850e006a2a5-kube-api-access-ndlg5\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:06.195925 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.195894    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-whb7g"
Sep 16 15:52:06.249999 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.249972    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktjg8\" (UniqueName: \"kubernetes.io/projected/d89c5398-15a5-4797-9759-63216417b9a4-kube-api-access-ktjg8\") pod \"service-ca-operator-cf7c776b7-65p44\" (UID: \"d89c5398-15a5-4797-9759-63216417b9a4\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44"
Sep 16 15:52:06.250123 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.250045    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d89c5398-15a5-4797-9759-63216417b9a4-config\") pod \"service-ca-operator-cf7c776b7-65p44\" (UID: \"d89c5398-15a5-4797-9759-63216417b9a4\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44"
Sep 16 15:52:06.250123 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.250101    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d89c5398-15a5-4797-9759-63216417b9a4-serving-cert\") pod \"service-ca-operator-cf7c776b7-65p44\" (UID: \"d89c5398-15a5-4797-9759-63216417b9a4\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44"
Sep 16 15:52:06.250909 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.250883    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d89c5398-15a5-4797-9759-63216417b9a4-config\") pod \"service-ca-operator-cf7c776b7-65p44\" (UID: \"d89c5398-15a5-4797-9759-63216417b9a4\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44"
Sep 16 15:52:06.252706 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.252678    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d89c5398-15a5-4797-9759-63216417b9a4-serving-cert\") pod \"service-ca-operator-cf7c776b7-65p44\" (UID: \"d89c5398-15a5-4797-9759-63216417b9a4\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44"
Sep 16 15:52:06.258365 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.258344    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktjg8\" (UniqueName: \"kubernetes.io/projected/d89c5398-15a5-4797-9759-63216417b9a4-kube-api-access-ktjg8\") pod \"service-ca-operator-cf7c776b7-65p44\" (UID: \"d89c5398-15a5-4797-9759-63216417b9a4\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44"
Sep 16 15:52:06.406130 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.406050    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44"
Sep 16 15:52:06.465020 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.464989    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-whb7g"]
Sep 16 15:52:06.468259 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:06.468232    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29cf843e_24ce_4d87_baea_993e52f3d2e4.slice/crio-2440d941e45cd31cd5dbcef867eecf0767441b9d1b9a826978a282bc0aea8be7 WatchSource:0}: Error finding container 2440d941e45cd31cd5dbcef867eecf0767441b9d1b9a826978a282bc0aea8be7: Status 404 returned error can't find the container with id 2440d941e45cd31cd5dbcef867eecf0767441b9d1b9a826978a282bc0aea8be7
Sep 16 15:52:06.653528 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.653484    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:06.653704 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.653569    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.653704 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.653613    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:06.653704 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:06.653653    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 15:52:06.653806 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:06.653701    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 15:52:06.653806 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:06.653716    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle podName:3b6415b2-6e76-454e-b85e-4e7e80cbd01e nodeName:}" failed. No retries permitted until 2026-09-16 15:52:07.653702188 +0000 UTC m=+124.173324948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle") pod "router-default-79955bb76b-kbcfl" (UID: "3b6415b2-6e76-454e-b85e-4e7e80cbd01e") : configmap references non-existent config key: service-ca.crt
Sep 16 15:52:06.653806 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:06.653731    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls podName:591b0e66-93df-447a-8ddd-d850e006a2a5 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:07.653724774 +0000 UTC m=+124.173347529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-28r4q" (UID: "591b0e66-93df-447a-8ddd-d850e006a2a5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 15:52:06.653806 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:06.653753    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs podName:3b6415b2-6e76-454e-b85e-4e7e80cbd01e nodeName:}" failed. No retries permitted until 2026-09-16 15:52:07.653735821 +0000 UTC m=+124.173358577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs") pod "router-default-79955bb76b-kbcfl" (UID: "3b6415b2-6e76-454e-b85e-4e7e80cbd01e") : secret "router-metrics-certs-default" not found
Sep 16 15:52:06.671983 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:06.671914    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44"]
Sep 16 15:52:06.674710 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:06.674689    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89c5398_15a5_4797_9759_63216417b9a4.slice/crio-c04669cdea7322760b10f84ed6daf1637a3bb9fa72e2ddf4c695608caaaf98b5 WatchSource:0}: Error finding container c04669cdea7322760b10f84ed6daf1637a3bb9fa72e2ddf4c695608caaaf98b5: Status 404 returned error can't find the container with id c04669cdea7322760b10f84ed6daf1637a3bb9fa72e2ddf4c695608caaaf98b5
Sep 16 15:52:07.437590 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:07.437553    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44" event={"ID":"d89c5398-15a5-4797-9759-63216417b9a4","Type":"ContainerStarted","Data":"c04669cdea7322760b10f84ed6daf1637a3bb9fa72e2ddf4c695608caaaf98b5"}
Sep 16 15:52:07.438917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:07.438879    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-whb7g" event={"ID":"29cf843e-24ce-4d87-baea-993e52f3d2e4","Type":"ContainerStarted","Data":"2440d941e45cd31cd5dbcef867eecf0767441b9d1b9a826978a282bc0aea8be7"}
Sep 16 15:52:07.660854 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:07.660816    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:07.661021 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:07.660867    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:07.661021 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:07.660919    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:07.661138 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:07.661027    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle podName:3b6415b2-6e76-454e-b85e-4e7e80cbd01e nodeName:}" failed. No retries permitted until 2026-09-16 15:52:09.661003121 +0000 UTC m=+126.180625880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle") pod "router-default-79955bb76b-kbcfl" (UID: "3b6415b2-6e76-454e-b85e-4e7e80cbd01e") : configmap references non-existent config key: service-ca.crt
Sep 16 15:52:07.661138 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:07.661047    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 15:52:07.661138 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:07.661108    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls podName:591b0e66-93df-447a-8ddd-d850e006a2a5 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:09.661093007 +0000 UTC m=+126.180715766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-28r4q" (UID: "591b0e66-93df-447a-8ddd-d850e006a2a5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 15:52:07.661138 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:07.661048    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 15:52:07.661314 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:07.661153    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs podName:3b6415b2-6e76-454e-b85e-4e7e80cbd01e nodeName:}" failed. No retries permitted until 2026-09-16 15:52:09.661141177 +0000 UTC m=+126.180763945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs") pod "router-default-79955bb76b-kbcfl" (UID: "3b6415b2-6e76-454e-b85e-4e7e80cbd01e") : secret "router-metrics-certs-default" not found
Sep 16 15:52:09.445567 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:09.445509    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44" event={"ID":"d89c5398-15a5-4797-9759-63216417b9a4","Type":"ContainerStarted","Data":"e1213fe024128987d04e5d7304b9acf3db3d8274b71565f7e6cf72d3bd7d8542"}
Sep 16 15:52:09.447289 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:09.447262    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-whb7g" event={"ID":"29cf843e-24ce-4d87-baea-993e52f3d2e4","Type":"ContainerStarted","Data":"0d7cda8c12bb7c1d51085b9390bbf420a0a1163daf5f941822b335398ac0d1a8"}
Sep 16 15:52:09.462015 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:09.461971    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44" podStartSLOduration=0.89336002 podStartE2EDuration="3.461958923s" totalImagesPullingTime="2.568598903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:09.461206947 +0000 UTC m=+125.980829722" watchObservedRunningTime="2026-09-16 15:52:09.461958923 +0000 UTC m=+125.981581700"
Sep 16 15:52:09.479815 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:09.479772    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-whb7g" podStartSLOduration=1.709287716 podStartE2EDuration="4.479756804s" totalImagesPullingTime="2.770469088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:09.479198505 +0000 UTC m=+125.998821283" watchObservedRunningTime="2026-09-16 15:52:09.479756804 +0000 UTC m=+125.999379581"
Sep 16 15:52:09.684016 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:09.683972    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:09.684195 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:09.684053    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:09.684195 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:09.684095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:09.684195 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:09.684111    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 15:52:09.684313 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:09.684200    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls podName:591b0e66-93df-447a-8ddd-d850e006a2a5 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:13.684177777 +0000 UTC m=+130.203800540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-28r4q" (UID: "591b0e66-93df-447a-8ddd-d850e006a2a5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 15:52:09.684313 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:09.684216    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 15:52:09.684313 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:09.684236    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle podName:3b6415b2-6e76-454e-b85e-4e7e80cbd01e nodeName:}" failed. No retries permitted until 2026-09-16 15:52:13.684220542 +0000 UTC m=+130.203843316 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle") pod "router-default-79955bb76b-kbcfl" (UID: "3b6415b2-6e76-454e-b85e-4e7e80cbd01e") : configmap references non-existent config key: service-ca.crt
Sep 16 15:52:09.684313 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:09.684271    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs podName:3b6415b2-6e76-454e-b85e-4e7e80cbd01e nodeName:}" failed. No retries permitted until 2026-09-16 15:52:13.684260572 +0000 UTC m=+130.203883328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs") pod "router-default-79955bb76b-kbcfl" (UID: "3b6415b2-6e76-454e-b85e-4e7e80cbd01e") : secret "router-metrics-certs-default" not found
Sep 16 15:52:11.068464 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.068429    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c6rkl"]
Sep 16 15:52:11.071830 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.071810    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.073953 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.073929    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 15:52:11.073953 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.073950    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ckzgw\""
Sep 16 15:52:11.074138 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.073995    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 15:52:11.082031 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.082008    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c6rkl"]
Sep 16 15:52:11.195405 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.195380    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/acac2a59-951e-427a-96ee-52e273629779-crio-socket\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.195588 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.195426    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.195588 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.195452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/acac2a59-951e-427a-96ee-52e273629779-data-volume\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.195588 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.195471    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.195588 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.195533    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2jgn\" (UniqueName: \"kubernetes.io/projected/acac2a59-951e-427a-96ee-52e273629779-kube-api-access-d2jgn\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.296759 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.296719    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2jgn\" (UniqueName: \"kubernetes.io/projected/acac2a59-951e-427a-96ee-52e273629779-kube-api-access-d2jgn\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.296895 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.296795    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/acac2a59-951e-427a-96ee-52e273629779-crio-socket\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.296895 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.296831    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.296895 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.296851    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/acac2a59-951e-427a-96ee-52e273629779-data-volume\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.296895 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.296869    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.297064 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.296938    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/acac2a59-951e-427a-96ee-52e273629779-crio-socket\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.297064 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:11.296962    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:11.297064 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:11.297002    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 15:52:11.297064 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:11.297016    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls podName:acac2a59-951e-427a-96ee-52e273629779 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:11.797002208 +0000 UTC m=+128.316624963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6rkl" (UID: "acac2a59-951e-427a-96ee-52e273629779") : secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:11.297187 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:11.297083    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm podName:acac2a59-951e-427a-96ee-52e273629779 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:11.797066438 +0000 UTC m=+128.316689198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6rkl" (UID: "acac2a59-951e-427a-96ee-52e273629779") : configmap "kube-rbac-proxy" not found
Sep 16 15:52:11.297265 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.297248    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/acac2a59-951e-427a-96ee-52e273629779-data-volume\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.305480 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.305448    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2jgn\" (UniqueName: \"kubernetes.io/projected/acac2a59-951e-427a-96ee-52e273629779-kube-api-access-d2jgn\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.564773 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.564737    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-kglsn"]
Sep 16 15:52:11.567311 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.567295    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kglsn"
Sep 16 15:52:11.569326 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.569302    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 15:52:11.569429 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.569403    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 15:52:11.569473 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.569421    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6fbkn\""
Sep 16 15:52:11.574783 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.574759    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-kglsn"]
Sep 16 15:52:11.700571 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.700514    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds62x\" (UniqueName: \"kubernetes.io/projected/0c163fdb-43f8-4589-bf57-c156d14ad5ca-kube-api-access-ds62x\") pod \"migrator-8b4d697c5-kglsn\" (UID: \"0c163fdb-43f8-4589-bf57-c156d14ad5ca\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kglsn"
Sep 16 15:52:11.800986 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.800948    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.800986 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.800989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds62x\" (UniqueName: \"kubernetes.io/projected/0c163fdb-43f8-4589-bf57-c156d14ad5ca-kube-api-access-ds62x\") pod \"migrator-8b4d697c5-kglsn\" (UID: \"0c163fdb-43f8-4589-bf57-c156d14ad5ca\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kglsn"
Sep 16 15:52:11.801279 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:11.801094    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:11.801279 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.801115    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:11.801279 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:11.801163    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls podName:acac2a59-951e-427a-96ee-52e273629779 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:12.801142121 +0000 UTC m=+129.320764876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6rkl" (UID: "acac2a59-951e-427a-96ee-52e273629779") : secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:11.801279 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:11.801176    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 15:52:11.801279 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:11.801219    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm podName:acac2a59-951e-427a-96ee-52e273629779 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:12.801204743 +0000 UTC m=+129.320827498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6rkl" (UID: "acac2a59-951e-427a-96ee-52e273629779") : configmap "kube-rbac-proxy" not found
Sep 16 15:52:11.810003 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.809984    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds62x\" (UniqueName: \"kubernetes.io/projected/0c163fdb-43f8-4589-bf57-c156d14ad5ca-kube-api-access-ds62x\") pod \"migrator-8b4d697c5-kglsn\" (UID: \"0c163fdb-43f8-4589-bf57-c156d14ad5ca\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kglsn"
Sep 16 15:52:11.876228 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:11.876204    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kglsn"
Sep 16 15:52:12.143750 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:12.140841    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-kglsn"]
Sep 16 15:52:12.147379 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:12.147349    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c163fdb_43f8_4589_bf57_c156d14ad5ca.slice/crio-7fb423f2aaa18e62b1d015c5fe134ef99957428d26cebc6e370e2200ba9deaad WatchSource:0}: Error finding container 7fb423f2aaa18e62b1d015c5fe134ef99957428d26cebc6e370e2200ba9deaad: Status 404 returned error can't find the container with id 7fb423f2aaa18e62b1d015c5fe134ef99957428d26cebc6e370e2200ba9deaad
Sep 16 15:52:12.456509 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:12.456436    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kglsn" event={"ID":"0c163fdb-43f8-4589-bf57-c156d14ad5ca","Type":"ContainerStarted","Data":"7fb423f2aaa18e62b1d015c5fe134ef99957428d26cebc6e370e2200ba9deaad"}
Sep 16 15:52:12.810019 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:12.809923    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:12.810019 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:12.809993    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:12.810229 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:12.810076    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:12.810229 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:12.810101    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 15:52:12.810229 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:12.810144    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls podName:acac2a59-951e-427a-96ee-52e273629779 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:14.810128216 +0000 UTC m=+131.329750971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6rkl" (UID: "acac2a59-951e-427a-96ee-52e273629779") : secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:12.810229 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:12.810161    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm podName:acac2a59-951e-427a-96ee-52e273629779 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:14.81015455 +0000 UTC m=+131.329777304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6rkl" (UID: "acac2a59-951e-427a-96ee-52e273629779") : configmap "kube-rbac-proxy" not found
Sep 16 15:52:12.829255 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:12.829233    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6r28d_eedc6ccd-a007-4ca4-98ba-54fa62367075/dns-node-resolver/0.log"
Sep 16 15:52:13.018034 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.018006    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-7bdlp"]
Sep 16 15:52:13.021080 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.021058    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-7bdlp"
Sep 16 15:52:13.023147 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.023124    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 15:52:13.023237 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.023212    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 15:52:13.023307 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.023212    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 15:52:13.023984 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.023967    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 15:52:13.024065 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.023972    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mqjgt\""
Sep 16 15:52:13.030811 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.030788    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-7bdlp"]
Sep 16 15:52:13.112326 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.112289    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2ca316c-e208-4a23-88f8-13beedb93300-signing-key\") pod \"service-ca-568869fdc5-7bdlp\" (UID: \"e2ca316c-e208-4a23-88f8-13beedb93300\") " pod="openshift-service-ca/service-ca-568869fdc5-7bdlp"
Sep 16 15:52:13.112472 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.112380    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dscx\" (UniqueName: \"kubernetes.io/projected/e2ca316c-e208-4a23-88f8-13beedb93300-kube-api-access-8dscx\") pod \"service-ca-568869fdc5-7bdlp\" (UID: \"e2ca316c-e208-4a23-88f8-13beedb93300\") " pod="openshift-service-ca/service-ca-568869fdc5-7bdlp"
Sep 16 15:52:13.112472 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.112455    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2ca316c-e208-4a23-88f8-13beedb93300-signing-cabundle\") pod \"service-ca-568869fdc5-7bdlp\" (UID: \"e2ca316c-e208-4a23-88f8-13beedb93300\") " pod="openshift-service-ca/service-ca-568869fdc5-7bdlp"
Sep 16 15:52:13.213161 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.213124    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2ca316c-e208-4a23-88f8-13beedb93300-signing-key\") pod \"service-ca-568869fdc5-7bdlp\" (UID: \"e2ca316c-e208-4a23-88f8-13beedb93300\") " pod="openshift-service-ca/service-ca-568869fdc5-7bdlp"
Sep 16 15:52:13.213576 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.213187    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dscx\" (UniqueName: \"kubernetes.io/projected/e2ca316c-e208-4a23-88f8-13beedb93300-kube-api-access-8dscx\") pod \"service-ca-568869fdc5-7bdlp\" (UID: \"e2ca316c-e208-4a23-88f8-13beedb93300\") " pod="openshift-service-ca/service-ca-568869fdc5-7bdlp"
Sep 16 15:52:13.213576 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.213250    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2ca316c-e208-4a23-88f8-13beedb93300-signing-cabundle\") pod \"service-ca-568869fdc5-7bdlp\" (UID: \"e2ca316c-e208-4a23-88f8-13beedb93300\") " pod="openshift-service-ca/service-ca-568869fdc5-7bdlp"
Sep 16 15:52:13.213978 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.213956    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2ca316c-e208-4a23-88f8-13beedb93300-signing-cabundle\") pod \"service-ca-568869fdc5-7bdlp\" (UID: \"e2ca316c-e208-4a23-88f8-13beedb93300\") " pod="openshift-service-ca/service-ca-568869fdc5-7bdlp"
Sep 16 15:52:13.215763 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.215742    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2ca316c-e208-4a23-88f8-13beedb93300-signing-key\") pod \"service-ca-568869fdc5-7bdlp\" (UID: \"e2ca316c-e208-4a23-88f8-13beedb93300\") " pod="openshift-service-ca/service-ca-568869fdc5-7bdlp"
Sep 16 15:52:13.221629 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.221598    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dscx\" (UniqueName: \"kubernetes.io/projected/e2ca316c-e208-4a23-88f8-13beedb93300-kube-api-access-8dscx\") pod \"service-ca-568869fdc5-7bdlp\" (UID: \"e2ca316c-e208-4a23-88f8-13beedb93300\") " pod="openshift-service-ca/service-ca-568869fdc5-7bdlp"
Sep 16 15:52:13.331387 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.331357    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-7bdlp"
Sep 16 15:52:13.461649 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.461618    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kglsn" event={"ID":"0c163fdb-43f8-4589-bf57-c156d14ad5ca","Type":"ContainerStarted","Data":"3dba113c6477133fb34ad9b7abbac5c303bc0a23e842af346427e1b2ae644c9d"}
Sep 16 15:52:13.604230 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.604198    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-7bdlp"]
Sep 16 15:52:13.608347 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:13.608317    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ca316c_e208_4a23_88f8_13beedb93300.slice/crio-329211ebfccf38b8a0a398ff4507f22dcc0a41066a9e14b098eea9af0e713736 WatchSource:0}: Error finding container 329211ebfccf38b8a0a398ff4507f22dcc0a41066a9e14b098eea9af0e713736: Status 404 returned error can't find the container with id 329211ebfccf38b8a0a398ff4507f22dcc0a41066a9e14b098eea9af0e713736
Sep 16 15:52:13.716819 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.716785    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:13.716819 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.716829    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:52:13.717043 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.716846    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:13.717043 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:13.716871    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:13.717043 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:13.716951    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 15:52:13.717043 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:13.716972    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 15:52:13.717043 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:13.716986    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle podName:3b6415b2-6e76-454e-b85e-4e7e80cbd01e nodeName:}" failed. No retries permitted until 2026-09-16 15:52:21.716967441 +0000 UTC m=+138.236590217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle") pod "router-default-79955bb76b-kbcfl" (UID: "3b6415b2-6e76-454e-b85e-4e7e80cbd01e") : configmap references non-existent config key: service-ca.crt
Sep 16 15:52:13.717043 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:13.716951    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 15:52:13.717043 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:13.717007    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs podName:1332ae3f-698f-4d0e-9801-2aa87121a12b nodeName:}" failed. No retries permitted until 2026-09-16 15:54:15.71699801 +0000 UTC m=+252.236620767 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs") pod "network-metrics-daemon-bbfmm" (UID: "1332ae3f-698f-4d0e-9801-2aa87121a12b") : secret "metrics-daemon-secret" not found
Sep 16 15:52:13.717043 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:13.717020    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls podName:591b0e66-93df-447a-8ddd-d850e006a2a5 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:21.717013443 +0000 UTC m=+138.236636199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-28r4q" (UID: "591b0e66-93df-447a-8ddd-d850e006a2a5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 15:52:13.717043 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:13.717032    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs podName:3b6415b2-6e76-454e-b85e-4e7e80cbd01e nodeName:}" failed. No retries permitted until 2026-09-16 15:52:21.717025686 +0000 UTC m=+138.236648442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs") pod "router-default-79955bb76b-kbcfl" (UID: "3b6415b2-6e76-454e-b85e-4e7e80cbd01e") : secret "router-metrics-certs-default" not found
Sep 16 15:52:14.050995 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:14.050927    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nz7nc_12c61b9c-b819-4330-bcf8-05341ce39ac8/node-ca/0.log"
Sep 16 15:52:14.470427 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:14.470391    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kglsn" event={"ID":"0c163fdb-43f8-4589-bf57-c156d14ad5ca","Type":"ContainerStarted","Data":"acfe4a5b2d8fc52d2a3d1c347c88e3d9301cbf581edd49e090af207f794f5c60"}
Sep 16 15:52:14.471950 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:14.471920    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-7bdlp" event={"ID":"e2ca316c-e208-4a23-88f8-13beedb93300","Type":"ContainerStarted","Data":"de053103d474bca87652ae5f4254c5b3b290781b4460cb3aa8ff06efe42e5f2f"}
Sep 16 15:52:14.472055 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:14.471970    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-7bdlp" event={"ID":"e2ca316c-e208-4a23-88f8-13beedb93300","Type":"ContainerStarted","Data":"329211ebfccf38b8a0a398ff4507f22dcc0a41066a9e14b098eea9af0e713736"}
Sep 16 15:52:14.485898 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:14.485855    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-kglsn" podStartSLOduration=2.300803986 podStartE2EDuration="3.485842489s" totalImagesPullingTime="1.185038503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:14.484317622 +0000 UTC m=+131.003940400" watchObservedRunningTime="2026-09-16 15:52:14.485842489 +0000 UTC m=+131.005465273"
Sep 16 15:52:14.498314 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:14.498265    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-7bdlp" podStartSLOduration=1.498253385 podStartE2EDuration="1.498253385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:14.497861825 +0000 UTC m=+131.017484615" watchObservedRunningTime="2026-09-16 15:52:14.498253385 +0000 UTC m=+131.017876162"
Sep 16 15:52:14.825096 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:14.825013    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:14.825096 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:14.825064    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:14.825268 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:14.825153    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 15:52:14.825268 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:14.825165    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:14.825268 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:14.825205    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm podName:acac2a59-951e-427a-96ee-52e273629779 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:18.825191703 +0000 UTC m=+135.344814457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6rkl" (UID: "acac2a59-951e-427a-96ee-52e273629779") : configmap "kube-rbac-proxy" not found
Sep 16 15:52:14.825268 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:14.825220    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls podName:acac2a59-951e-427a-96ee-52e273629779 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:18.825213544 +0000 UTC m=+135.344836298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6rkl" (UID: "acac2a59-951e-427a-96ee-52e273629779") : secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:15.030763 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:15.030734    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-twc8n_1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59/serve-healthcheck-canary/0.log"
Sep 16 15:52:18.858394 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:18.858356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:18.858910 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:18.858414    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:18.858910 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:18.858482    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 15:52:18.858910 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:18.858535    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:18.858910 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:18.858551    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm podName:acac2a59-951e-427a-96ee-52e273629779 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:26.858534258 +0000 UTC m=+143.378157029 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c6rkl" (UID: "acac2a59-951e-427a-96ee-52e273629779") : configmap "kube-rbac-proxy" not found
Sep 16 15:52:18.858910 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:18.858603    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls podName:acac2a59-951e-427a-96ee-52e273629779 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:26.858588812 +0000 UTC m=+143.378211567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c6rkl" (UID: "acac2a59-951e-427a-96ee-52e273629779") : secret "insights-runtime-extractor-tls" not found
Sep 16 15:52:21.784905 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:21.784867    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:21.785396 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:21.784913    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:21.785396 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:21.784980    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:21.785396 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:21.785090    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 15:52:21.785396 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:21.785163    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls podName:591b0e66-93df-447a-8ddd-d850e006a2a5 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:37.785147824 +0000 UTC m=+154.304770578 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-28r4q" (UID: "591b0e66-93df-447a-8ddd-d850e006a2a5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 15:52:21.785700 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:21.785676    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-service-ca-bundle\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:21.787297 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:21.787280    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b6415b2-6e76-454e-b85e-4e7e80cbd01e-metrics-certs\") pod \"router-default-79955bb76b-kbcfl\" (UID: \"3b6415b2-6e76-454e-b85e-4e7e80cbd01e\") " pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:21.908278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:21.908236    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:22.179316 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:22.179285    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79955bb76b-kbcfl"]
Sep 16 15:52:22.182242 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:22.182218    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6415b2_6e76_454e_b85e_4e7e80cbd01e.slice/crio-0466f000ee1635b72fc08d4041e2ae1e2bfa96d262b7cbaa80f4498436e66dfa WatchSource:0}: Error finding container 0466f000ee1635b72fc08d4041e2ae1e2bfa96d262b7cbaa80f4498436e66dfa: Status 404 returned error can't find the container with id 0466f000ee1635b72fc08d4041e2ae1e2bfa96d262b7cbaa80f4498436e66dfa
Sep 16 15:52:22.499663 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:22.499585    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79955bb76b-kbcfl" event={"ID":"3b6415b2-6e76-454e-b85e-4e7e80cbd01e","Type":"ContainerStarted","Data":"039f34f39213ffae885defd80de084eaaa2ae81fa89e23c6f392d96e0f7eae39"}
Sep 16 15:52:22.499663 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:22.499643    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79955bb76b-kbcfl" event={"ID":"3b6415b2-6e76-454e-b85e-4e7e80cbd01e","Type":"ContainerStarted","Data":"0466f000ee1635b72fc08d4041e2ae1e2bfa96d262b7cbaa80f4498436e66dfa"}
Sep 16 15:52:22.516935 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:22.516884    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-79955bb76b-kbcfl" podStartSLOduration=17.516868153 podStartE2EDuration="17.516868153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:22.515936395 +0000 UTC m=+139.035559172" watchObservedRunningTime="2026-09-16 15:52:22.516868153 +0000 UTC m=+139.036490944"
Sep 16 15:52:22.908713 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:22.908674    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:22.911395 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:22.911372    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:23.502864 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:23.502837    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:23.504102 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:23.504079    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-79955bb76b-kbcfl"
Sep 16 15:52:26.924843 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:26.924804    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:26.925222 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:26.924854    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:26.925367 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:26.925333    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/acac2a59-951e-427a-96ee-52e273629779-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:26.927096 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:26.927076    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/acac2a59-951e-427a-96ee-52e273629779-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c6rkl\" (UID: \"acac2a59-951e-427a-96ee-52e273629779\") " pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:26.981281 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:26.981245    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c6rkl"
Sep 16 15:52:27.253409 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:27.253330    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c6rkl"]
Sep 16 15:52:27.257218 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:27.257189    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacac2a59_951e_427a_96ee_52e273629779.slice/crio-86f2bf0da5670f60a9b11f4a71c89e7b7b6e9de9c41324642f36b4f6857872ca WatchSource:0}: Error finding container 86f2bf0da5670f60a9b11f4a71c89e7b7b6e9de9c41324642f36b4f6857872ca: Status 404 returned error can't find the container with id 86f2bf0da5670f60a9b11f4a71c89e7b7b6e9de9c41324642f36b4f6857872ca
Sep 16 15:52:27.515200 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:27.515114    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c6rkl" event={"ID":"acac2a59-951e-427a-96ee-52e273629779","Type":"ContainerStarted","Data":"ed2396c84807085f95f7d54a0f911141c033f3e86557f8abfc92e85a73214889"}
Sep 16 15:52:27.515200 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:27.515172    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c6rkl" event={"ID":"acac2a59-951e-427a-96ee-52e273629779","Type":"ContainerStarted","Data":"86f2bf0da5670f60a9b11f4a71c89e7b7b6e9de9c41324642f36b4f6857872ca"}
Sep 16 15:52:28.520012 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:28.519983    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c6rkl" event={"ID":"acac2a59-951e-427a-96ee-52e273629779","Type":"ContainerStarted","Data":"2ef67b9dacaf8d0671db2c56789d7d30523200000a7cdd5c7ee9a50b4f4b3d33"}
Sep 16 15:52:30.528765 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:30.528722    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c6rkl" event={"ID":"acac2a59-951e-427a-96ee-52e273629779","Type":"ContainerStarted","Data":"cf6396c066c28f2adb72563120815438aebd71df076bef0ec3dbe00795aa3d61"}
Sep 16 15:52:30.548020 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:30.547973    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c6rkl" podStartSLOduration=17.293109346 podStartE2EDuration="19.547955401s" totalImagesPullingTime="2.254846055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:52:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:30.546378858 +0000 UTC m=+147.066001635" watchObservedRunningTime="2026-09-16 15:52:30.547955401 +0000 UTC m=+147.067578178"
Sep 16 15:52:37.808685 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:37.808634    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:37.811022 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:37.810994    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/591b0e66-93df-447a-8ddd-d850e006a2a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-28r4q\" (UID: \"591b0e66-93df-447a-8ddd-d850e006a2a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:38.102080 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:38.102049    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"
Sep 16 15:52:38.367858 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:38.367780    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q"]
Sep 16 15:52:38.370878 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:38.370847    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591b0e66_93df_447a_8ddd_d850e006a2a5.slice/crio-86e9562c80d76dffc65eb529ace585c9edc8f3c530973c2248d5fd6eaed36c3f WatchSource:0}: Error finding container 86e9562c80d76dffc65eb529ace585c9edc8f3c530973c2248d5fd6eaed36c3f: Status 404 returned error can't find the container with id 86e9562c80d76dffc65eb529ace585c9edc8f3c530973c2248d5fd6eaed36c3f
Sep 16 15:52:38.555830 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:38.555791    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q" event={"ID":"591b0e66-93df-447a-8ddd-d850e006a2a5","Type":"ContainerStarted","Data":"86e9562c80d76dffc65eb529ace585c9edc8f3c530973c2248d5fd6eaed36c3f"}
Sep 16 15:52:40.564287 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:40.564252    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q" event={"ID":"591b0e66-93df-447a-8ddd-d850e006a2a5","Type":"ContainerStarted","Data":"7022e30bdf9e915dffbac9d7e68ae3c189d2ad786b4270faafd61d76fefd867a"}
Sep 16 15:52:40.582115 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:40.582062    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-28r4q" podStartSLOduration=33.978296521 podStartE2EDuration="35.582042074s" totalImagesPullingTime="1.603745553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:40.581867185 +0000 UTC m=+157.101489966" watchObservedRunningTime="2026-09-16 15:52:40.582042074 +0000 UTC m=+157.101664851"
Sep 16 15:52:40.864081 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:40.864041    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wq8bz" podUID="cb25687b-a293-40ae-a3a7-e2357384bcd4"
Sep 16 15:52:41.014015 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:41.013952    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-bbfmm" podUID="1332ae3f-698f-4d0e-9801-2aa87121a12b"
Sep 16 15:52:41.567319 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:41.567285    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:52:45.768969 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:45.768924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:52:45.771238 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:45.771216    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb25687b-a293-40ae-a3a7-e2357384bcd4-metrics-tls\") pod \"dns-default-wq8bz\" (UID: \"cb25687b-a293-40ae-a3a7-e2357384bcd4\") " pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:52:46.070160 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:46.070076    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-thq8g\""
Sep 16 15:52:46.078865 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:46.078839    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:52:46.346498 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:46.346468    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wq8bz"]
Sep 16 15:52:46.349432 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:46.349407    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb25687b_a293_40ae_a3a7_e2357384bcd4.slice/crio-abbfa00810dbd6c67570d3140724a83cc2965162d0cca96c6830063cbeabeff8 WatchSource:0}: Error finding container abbfa00810dbd6c67570d3140724a83cc2965162d0cca96c6830063cbeabeff8: Status 404 returned error can't find the container with id abbfa00810dbd6c67570d3140724a83cc2965162d0cca96c6830063cbeabeff8
Sep 16 15:52:46.584583 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:46.584536    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wq8bz" event={"ID":"cb25687b-a293-40ae-a3a7-e2357384bcd4","Type":"ContainerStarted","Data":"abbfa00810dbd6c67570d3140724a83cc2965162d0cca96c6830063cbeabeff8"}
Sep 16 15:52:48.595109 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.595015    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wq8bz" event={"ID":"cb25687b-a293-40ae-a3a7-e2357384bcd4","Type":"ContainerStarted","Data":"20bebe3c9d7eae9d62fc53c4c305bf6c5f63bfabfc2b21b7d9969457651c340a"}
Sep 16 15:52:48.595109 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.595087    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wq8bz" event={"ID":"cb25687b-a293-40ae-a3a7-e2357384bcd4","Type":"ContainerStarted","Data":"6e83e7cb34063180e46206b5599198836dc03f407550b06c1bc00207d8d01d48"}
Sep 16 15:52:48.596582 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.596536    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:52:48.620830 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.620764    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wq8bz" podStartSLOduration=129.799497361 podStartE2EDuration="2m11.620741712s" totalImagesPullingTime="1.821244351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:48.619838681 +0000 UTC m=+165.139461459" watchObservedRunningTime="2026-09-16 15:52:48.620741712 +0000 UTC m=+165.140364490"
Sep 16 15:52:48.915131 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.915091    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"]
Sep 16 15:52:48.919368 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.919347    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:48.923214 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.922955    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 15:52:48.923214 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.922962    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 15:52:48.923214 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.922962    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-z47br\""
Sep 16 15:52:48.923851 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.923827    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 15:52:48.937222 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.937196    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"]
Sep 16 15:52:48.995778 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.995741    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e72a7c5c-e549-45e7-8c91-f1d04a832452-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-98x75\" (UID: \"e72a7c5c-e549-45e7-8c91-f1d04a832452\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:48.995977 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.995825    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e72a7c5c-e549-45e7-8c91-f1d04a832452-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-98x75\" (UID: \"e72a7c5c-e549-45e7-8c91-f1d04a832452\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:48.995977 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.995852    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9mdz\" (UniqueName: \"kubernetes.io/projected/e72a7c5c-e549-45e7-8c91-f1d04a832452-kube-api-access-s9mdz\") pod \"openshift-state-metrics-67dcf8c979-98x75\" (UID: \"e72a7c5c-e549-45e7-8c91-f1d04a832452\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:48.995977 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:48.995885    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72a7c5c-e549-45e7-8c91-f1d04a832452-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-98x75\" (UID: \"e72a7c5c-e549-45e7-8c91-f1d04a832452\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:49.012026 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.011992    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qslcd"]
Sep 16 15:52:49.016381 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.016362    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.019832 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.019809    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 15:52:49.019974 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.019851    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 15:52:49.020077 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.020056    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 15:52:49.020196 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.020154    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fzfxs\""
Sep 16 15:52:49.097090 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.097054    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e72a7c5c-e549-45e7-8c91-f1d04a832452-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-98x75\" (UID: \"e72a7c5c-e549-45e7-8c91-f1d04a832452\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:49.097263 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.097106    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-wtmp\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.097263 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.097141    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e7589c-0441-4224-a5f1-e80d4759d220-sys\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.097263 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.097171    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.097263 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.097205    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g92p8\" (UniqueName: \"kubernetes.io/projected/38e7589c-0441-4224-a5f1-e80d4759d220-kube-api-access-g92p8\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.097433 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.097305    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-textfile\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.097433 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.097346    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e72a7c5c-e549-45e7-8c91-f1d04a832452-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-98x75\" (UID: \"e72a7c5c-e549-45e7-8c91-f1d04a832452\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:49.097433 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.097373    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38e7589c-0441-4224-a5f1-e80d4759d220-root\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.097433 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.097401    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9mdz\" (UniqueName: \"kubernetes.io/projected/e72a7c5c-e549-45e7-8c91-f1d04a832452-kube-api-access-s9mdz\") pod \"openshift-state-metrics-67dcf8c979-98x75\" (UID: \"e72a7c5c-e549-45e7-8c91-f1d04a832452\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:49.097433 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.097428    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-tls\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.097681 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.097461    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72a7c5c-e549-45e7-8c91-f1d04a832452-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-98x75\" (UID: \"e72a7c5c-e549-45e7-8c91-f1d04a832452\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:49.097681 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.097485    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38e7589c-0441-4224-a5f1-e80d4759d220-metrics-client-ca\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.097681 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.097511    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-accelerators-collector-config\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.097681 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:49.097574    2574 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 16 15:52:49.097681 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:49.097644    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e72a7c5c-e549-45e7-8c91-f1d04a832452-openshift-state-metrics-tls podName:e72a7c5c-e549-45e7-8c91-f1d04a832452 nodeName:}" failed. No retries permitted until 2026-09-16 15:52:49.597621889 +0000 UTC m=+166.117244657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e72a7c5c-e549-45e7-8c91-f1d04a832452-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-98x75" (UID: "e72a7c5c-e549-45e7-8c91-f1d04a832452") : secret "openshift-state-metrics-tls" not found
Sep 16 15:52:49.097907 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.097887    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e72a7c5c-e549-45e7-8c91-f1d04a832452-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-98x75\" (UID: \"e72a7c5c-e549-45e7-8c91-f1d04a832452\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:49.100205 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.100186    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e72a7c5c-e549-45e7-8c91-f1d04a832452-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-98x75\" (UID: \"e72a7c5c-e549-45e7-8c91-f1d04a832452\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:49.109038 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.109009    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9mdz\" (UniqueName: \"kubernetes.io/projected/e72a7c5c-e549-45e7-8c91-f1d04a832452-kube-api-access-s9mdz\") pod \"openshift-state-metrics-67dcf8c979-98x75\" (UID: \"e72a7c5c-e549-45e7-8c91-f1d04a832452\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:49.198622 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.198503    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-wtmp\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.198622 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.198594    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e7589c-0441-4224-a5f1-e80d4759d220-sys\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.198837 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.198624    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.198837 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.198658    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g92p8\" (UniqueName: \"kubernetes.io/projected/38e7589c-0441-4224-a5f1-e80d4759d220-kube-api-access-g92p8\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.198837 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.198674    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-wtmp\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.198837 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.198687    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e7589c-0441-4224-a5f1-e80d4759d220-sys\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.198837 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.198693    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-textfile\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.198837 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.198764    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38e7589c-0441-4224-a5f1-e80d4759d220-root\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.198837 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.198794    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-tls\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.199178 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.198849    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38e7589c-0441-4224-a5f1-e80d4759d220-metrics-client-ca\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.199178 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.198877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-accelerators-collector-config\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.199178 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.199006    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-textfile\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.199404 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.199383    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38e7589c-0441-4224-a5f1-e80d4759d220-root\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.199641 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.199618    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-accelerators-collector-config\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.200081 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.200062    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38e7589c-0441-4224-a5f1-e80d4759d220-metrics-client-ca\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.201776 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.201748    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.201951 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.201933    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38e7589c-0441-4224-a5f1-e80d4759d220-node-exporter-tls\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.206596 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.206568    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g92p8\" (UniqueName: \"kubernetes.io/projected/38e7589c-0441-4224-a5f1-e80d4759d220-kube-api-access-g92p8\") pod \"node-exporter-qslcd\" (UID: \"38e7589c-0441-4224-a5f1-e80d4759d220\") " pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.327091 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.327053    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qslcd"
Sep 16 15:52:49.340357 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:49.340324    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e7589c_0441_4224_a5f1_e80d4759d220.slice/crio-3021408de596e2b4f8936e42dd5c29039831dc6e641d424520228f0bacd03696 WatchSource:0}: Error finding container 3021408de596e2b4f8936e42dd5c29039831dc6e641d424520228f0bacd03696: Status 404 returned error can't find the container with id 3021408de596e2b4f8936e42dd5c29039831dc6e641d424520228f0bacd03696
Sep 16 15:52:49.601738 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.601705    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qslcd" event={"ID":"38e7589c-0441-4224-a5f1-e80d4759d220","Type":"ContainerStarted","Data":"3021408de596e2b4f8936e42dd5c29039831dc6e641d424520228f0bacd03696"}
Sep 16 15:52:49.602114 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.602094    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72a7c5c-e549-45e7-8c91-f1d04a832452-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-98x75\" (UID: \"e72a7c5c-e549-45e7-8c91-f1d04a832452\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:49.604353 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.604331    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72a7c5c-e549-45e7-8c91-f1d04a832452-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-98x75\" (UID: \"e72a7c5c-e549-45e7-8c91-f1d04a832452\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:49.829715 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:49.829673    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"
Sep 16 15:52:50.025994 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.025964    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 15:52:50.030332 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.030305    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.032788 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.032761    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 15:52:50.032788 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.032774    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 15:52:50.033110 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.033089    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 15:52:50.033221 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.033113    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 15:52:50.033286 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.033233    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 15:52:50.033376 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.033291    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 15:52:50.033628 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.033599    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 15:52:50.033628 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.033617    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 15:52:50.033763 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.033640    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 15:52:50.033808 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.033755    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lvf5g\""
Sep 16 15:52:50.045959 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.045935    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 15:52:50.101710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.101672    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75"]
Sep 16 15:52:50.104826 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:50.104794    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72a7c5c_e549_45e7_8c91_f1d04a832452.slice/crio-3a88cb01a48978ffa56fa830590c2f07f3e2c90cf461d546bee02e0c054f26cd WatchSource:0}: Error finding container 3a88cb01a48978ffa56fa830590c2f07f3e2c90cf461d546bee02e0c054f26cd: Status 404 returned error can't find the container with id 3a88cb01a48978ffa56fa830590c2f07f3e2c90cf461d546bee02e0c054f26cd
Sep 16 15:52:50.106799 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.106757    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.106863 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.106804    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/916482a0-583c-40c2-96a9-f97a21c8652f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.106863 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.106829    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.106863 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.106859    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.107036 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.106882    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-web-config\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.107036 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.106948    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.107036 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.106995    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/916482a0-583c-40c2-96a9-f97a21c8652f-config-out\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.107036 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.107023    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.107242 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.107069    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.107242 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.107094    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.107242 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.107154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gsjt\" (UniqueName: \"kubernetes.io/projected/916482a0-583c-40c2-96a9-f97a21c8652f-kube-api-access-2gsjt\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.107242 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.107188    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.107438 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.107275    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-config-volume\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.208622 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.208586    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.208764 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.208650    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/916482a0-583c-40c2-96a9-f97a21c8652f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.208764 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.208675    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.209069 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.209051    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.209228 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.209193    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.209290 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.209245    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-web-config\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.209290 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.209278    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.209397 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.209322    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/916482a0-583c-40c2-96a9-f97a21c8652f-config-out\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.209397 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.209347    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.209492 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.209403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.209492 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.209428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.209492 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.209467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gsjt\" (UniqueName: \"kubernetes.io/projected/916482a0-583c-40c2-96a9-f97a21c8652f-kube-api-access-2gsjt\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.209673 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.209493    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.209673 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.209569    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-config-volume\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.210591 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.210567    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.212267 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.211779    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.212267 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.212099    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/916482a0-583c-40c2-96a9-f97a21c8652f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.212267 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:52:50.212216    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-trusted-ca-bundle podName:916482a0-583c-40c2-96a9-f97a21c8652f nodeName:}" failed. No retries permitted until 2026-09-16 15:52:50.712195254 +0000 UTC m=+167.231818021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "916482a0-583c-40c2-96a9-f97a21c8652f") : configmap references non-existent config key: ca-bundle.crt
Sep 16 15:52:50.213078 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.213034    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-config-volume\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.213992 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.213939    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.215360 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.215318    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/916482a0-583c-40c2-96a9-f97a21c8652f-config-out\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.215631 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.215606    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.215960 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.215916    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-web-config\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.216111 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.216052    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.216604 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.216568    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.220644 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.220601    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gsjt\" (UniqueName: \"kubernetes.io/projected/916482a0-583c-40c2-96a9-f97a21c8652f-kube-api-access-2gsjt\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.608167 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.608133    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75" event={"ID":"e72a7c5c-e549-45e7-8c91-f1d04a832452","Type":"ContainerStarted","Data":"7df4835d8ddca2dada4ba78f6e11213b53080849483d54372a8e2c5cadf28a33"}
Sep 16 15:52:50.608550 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.608211    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75" event={"ID":"e72a7c5c-e549-45e7-8c91-f1d04a832452","Type":"ContainerStarted","Data":"3eb298cfac8efd4b49f7d74e05b8c91b2a8bf54cbcef6c381bc85417184653e8"}
Sep 16 15:52:50.608550 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.608243    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75" event={"ID":"e72a7c5c-e549-45e7-8c91-f1d04a832452","Type":"ContainerStarted","Data":"3a88cb01a48978ffa56fa830590c2f07f3e2c90cf461d546bee02e0c054f26cd"}
Sep 16 15:52:50.713222 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.713183    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.713957 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.713933    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.906691 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.906658    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-979kp"]
Sep 16 15:52:50.909953 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.909920    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-979kp"
Sep 16 15:52:50.912335 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.912311    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 15:52:50.912335 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.912324    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 15:52:50.912533 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.912334    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-nmddt\""
Sep 16 15:52:50.921742 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.921717    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-979kp"]
Sep 16 15:52:50.940737 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.940708    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:52:50.983800 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.982819    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"]
Sep 16 15:52:50.988199 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.988145    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:50.990868 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.990842    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 15:52:50.991036 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.991014    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 15:52:50.991108 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.991064    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 15:52:50.991250 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.991229    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 15:52:50.991351 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.991232    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dqzn9\""
Sep 16 15:52:50.991472 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.991335    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 15:52:50.991676 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:50.991649    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-edhtli2o2gje4\""
Sep 16 15:52:51.008288 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.008191    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"]
Sep 16 15:52:51.015653 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.015625    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qz8d\" (UniqueName: \"kubernetes.io/projected/15e1a31e-0ac4-4e41-9d1a-be65065d6089-kube-api-access-7qz8d\") pod \"downloads-76f99584f4-979kp\" (UID: \"15e1a31e-0ac4-4e41-9d1a-be65065d6089\") " pod="openshift-console/downloads-76f99584f4-979kp"
Sep 16 15:52:51.015796 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.015665    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2781fc36-78d3-4e86-b25c-6e99a11328fb-metrics-client-ca\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.015796 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.015688    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.015796 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.015745    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xtml\" (UniqueName: \"kubernetes.io/projected/2781fc36-78d3-4e86-b25c-6e99a11328fb-kube-api-access-4xtml\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.015964 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.015864    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.015964 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.015907    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.016041 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.015960    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-tls\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.016041 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.016017    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-grpc-tls\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.016108 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.016044    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.117147 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.117103    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-grpc-tls\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.117301 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.117167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.117301 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.117199    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qz8d\" (UniqueName: \"kubernetes.io/projected/15e1a31e-0ac4-4e41-9d1a-be65065d6089-kube-api-access-7qz8d\") pod \"downloads-76f99584f4-979kp\" (UID: \"15e1a31e-0ac4-4e41-9d1a-be65065d6089\") " pod="openshift-console/downloads-76f99584f4-979kp"
Sep 16 15:52:51.117301 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.117234    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2781fc36-78d3-4e86-b25c-6e99a11328fb-metrics-client-ca\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.117301 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.117264    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.117301 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.117295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xtml\" (UniqueName: \"kubernetes.io/projected/2781fc36-78d3-4e86-b25c-6e99a11328fb-kube-api-access-4xtml\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.117602 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.117348    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.117602 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.117382    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.117602 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.117414    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-tls\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.118205 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.118044    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2781fc36-78d3-4e86-b25c-6e99a11328fb-metrics-client-ca\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.120199 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.120149    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-grpc-tls\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.120827 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.120362    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.120827 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.120362    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.120827 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.120674    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.120827 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.120783    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.121131 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.121110    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2781fc36-78d3-4e86-b25c-6e99a11328fb-secret-thanos-querier-tls\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.124896 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.124873    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xtml\" (UniqueName: \"kubernetes.io/projected/2781fc36-78d3-4e86-b25c-6e99a11328fb-kube-api-access-4xtml\") pod \"thanos-querier-57cb8f89c6-7h58l\" (UID: \"2781fc36-78d3-4e86-b25c-6e99a11328fb\") " pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.125061 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.125045    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qz8d\" (UniqueName: \"kubernetes.io/projected/15e1a31e-0ac4-4e41-9d1a-be65065d6089-kube-api-access-7qz8d\") pod \"downloads-76f99584f4-979kp\" (UID: \"15e1a31e-0ac4-4e41-9d1a-be65065d6089\") " pod="openshift-console/downloads-76f99584f4-979kp"
Sep 16 15:52:51.221020 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.220933    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-979kp"
Sep 16 15:52:51.229006 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.228976    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 15:52:51.295595 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:51.295552    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916482a0_583c_40c2_96a9_f97a21c8652f.slice/crio-48593ea17fcde2bc0967500ee224be70bdbf3090fb85a384f8e84726469c3382 WatchSource:0}: Error finding container 48593ea17fcde2bc0967500ee224be70bdbf3090fb85a384f8e84726469c3382: Status 404 returned error can't find the container with id 48593ea17fcde2bc0967500ee224be70bdbf3090fb85a384f8e84726469c3382
Sep 16 15:52:51.301225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.301200    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:51.570600 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.570502    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-979kp"]
Sep 16 15:52:51.573792 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:51.573754    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e1a31e_0ac4_4e41_9d1a_be65065d6089.slice/crio-8b22ef9f70c454307ab6374f23ff3a72e3952ff257f152d4b9a6c767cf680c84 WatchSource:0}: Error finding container 8b22ef9f70c454307ab6374f23ff3a72e3952ff257f152d4b9a6c767cf680c84: Status 404 returned error can't find the container with id 8b22ef9f70c454307ab6374f23ff3a72e3952ff257f152d4b9a6c767cf680c84
Sep 16 15:52:51.590786 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.590705    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"]
Sep 16 15:52:51.594427 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:51.594394    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2781fc36_78d3_4e86_b25c_6e99a11328fb.slice/crio-3e6b3a02a126e8596591aa3d63fb3aa809e6590550b53e82da9676284efcc38c WatchSource:0}: Error finding container 3e6b3a02a126e8596591aa3d63fb3aa809e6590550b53e82da9676284efcc38c: Status 404 returned error can't find the container with id 3e6b3a02a126e8596591aa3d63fb3aa809e6590550b53e82da9676284efcc38c
Sep 16 15:52:51.615249 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.615218    2574 generic.go:356] "Generic (PLEG): container finished" podID="38e7589c-0441-4224-a5f1-e80d4759d220" containerID="ad0612322e297aae660e7da0bd6a2250d0086fa4b081950ac710b760536ddf01" exitCode=0
Sep 16 15:52:51.615656 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.615266    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qslcd" event={"ID":"38e7589c-0441-4224-a5f1-e80d4759d220","Type":"ContainerDied","Data":"ad0612322e297aae660e7da0bd6a2250d0086fa4b081950ac710b760536ddf01"}
Sep 16 15:52:51.616573 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.616552    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l" event={"ID":"2781fc36-78d3-4e86-b25c-6e99a11328fb","Type":"ContainerStarted","Data":"3e6b3a02a126e8596591aa3d63fb3aa809e6590550b53e82da9676284efcc38c"}
Sep 16 15:52:51.617815 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.617785    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-979kp" event={"ID":"15e1a31e-0ac4-4e41-9d1a-be65065d6089","Type":"ContainerStarted","Data":"8b22ef9f70c454307ab6374f23ff3a72e3952ff257f152d4b9a6c767cf680c84"}
Sep 16 15:52:51.619957 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.619930    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75" event={"ID":"e72a7c5c-e549-45e7-8c91-f1d04a832452","Type":"ContainerStarted","Data":"3c64383ff52b4ae72c92f7a2f989ae0d2b1609986396a3d03b96d6708391ae8b"}
Sep 16 15:52:51.621173 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:51.621152    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerStarted","Data":"48593ea17fcde2bc0967500ee224be70bdbf3090fb85a384f8e84726469c3382"}
Sep 16 15:52:52.627507 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:52.627467    2574 generic.go:356] "Generic (PLEG): container finished" podID="916482a0-583c-40c2-96a9-f97a21c8652f" containerID="da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13" exitCode=0
Sep 16 15:52:52.628017 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:52.627556    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerDied","Data":"da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13"}
Sep 16 15:52:52.630445 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:52.630414    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qslcd" event={"ID":"38e7589c-0441-4224-a5f1-e80d4759d220","Type":"ContainerStarted","Data":"029fc3aa8e0c0f4a41914c9de549dfe7c3b3409e8ad043762024cd2b605a7f38"}
Sep 16 15:52:52.630593 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:52.630477    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qslcd" event={"ID":"38e7589c-0441-4224-a5f1-e80d4759d220","Type":"ContainerStarted","Data":"005548b4ba6f5d21c09f792d0c608686e2b9a3226e303eb75606f3c6f8dd0540"}
Sep 16 15:52:52.658888 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:52.658832    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-98x75" podStartSLOduration=3.575014123 podStartE2EDuration="4.658798143s" totalImagesPullingTime="1.08378402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:51.648028818 +0000 UTC m=+168.167651609" watchObservedRunningTime="2026-09-16 15:52:52.658798143 +0000 UTC m=+169.178420921"
Sep 16 15:52:52.676760 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:52.676334    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qslcd" podStartSLOduration=3.404135602 podStartE2EDuration="4.676317506s" totalImagesPullingTime="1.207291028s" totalInitContainerRuntime="64.890876ms" isStatefulPod=true podCreationTimestamp="2026-09-16 15:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:52.67525058 +0000 UTC m=+169.194873368" watchObservedRunningTime="2026-09-16 15:52:52.676317506 +0000 UTC m=+169.195940288"
Sep 16 15:52:53.258885 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.258847    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-66568f976d-rkd2f"]
Sep 16 15:52:53.262414 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.262366    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.264657 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.264629    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 15:52:53.264769 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.264629    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ghb9f\""
Sep 16 15:52:53.265295 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.265274    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 15:52:53.265424 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.265371    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 15:52:53.265601 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.265587    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 15:52:53.265706 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.265665    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-70sgg0fi67o7u\""
Sep 16 15:52:53.270357 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.270331    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66568f976d-rkd2f"]
Sep 16 15:52:53.336045 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.335880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6962731d-8a32-42c9-8cac-e137bef90b68-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.336045 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.335934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6962731d-8a32-42c9-8cac-e137bef90b68-audit-log\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.336045 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.335968    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6962731d-8a32-42c9-8cac-e137bef90b68-client-ca-bundle\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.336045 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.336001    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6962731d-8a32-42c9-8cac-e137bef90b68-secret-metrics-server-tls\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.336402 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.336156    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6962731d-8a32-42c9-8cac-e137bef90b68-secret-metrics-server-client-certs\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.336402 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.336219    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6962731d-8a32-42c9-8cac-e137bef90b68-metrics-server-audit-profiles\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.336402 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.336244    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2d66\" (UniqueName: \"kubernetes.io/projected/6962731d-8a32-42c9-8cac-e137bef90b68-kube-api-access-k2d66\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.438686 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.437434    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6962731d-8a32-42c9-8cac-e137bef90b68-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.438686 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.437477    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6962731d-8a32-42c9-8cac-e137bef90b68-audit-log\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.438686 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.437501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6962731d-8a32-42c9-8cac-e137bef90b68-client-ca-bundle\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.438686 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.437552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6962731d-8a32-42c9-8cac-e137bef90b68-secret-metrics-server-tls\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.438686 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.437651    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6962731d-8a32-42c9-8cac-e137bef90b68-secret-metrics-server-client-certs\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.438686 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.437692    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6962731d-8a32-42c9-8cac-e137bef90b68-metrics-server-audit-profiles\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.438686 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.437715    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2d66\" (UniqueName: \"kubernetes.io/projected/6962731d-8a32-42c9-8cac-e137bef90b68-kube-api-access-k2d66\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.438686 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.438256    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6962731d-8a32-42c9-8cac-e137bef90b68-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.438686 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.438340    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6962731d-8a32-42c9-8cac-e137bef90b68-audit-log\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.439371 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.439341    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6962731d-8a32-42c9-8cac-e137bef90b68-metrics-server-audit-profiles\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.441748 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.441472    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6962731d-8a32-42c9-8cac-e137bef90b68-client-ca-bundle\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.441748 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.441707    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6962731d-8a32-42c9-8cac-e137bef90b68-secret-metrics-server-client-certs\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.441941 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.441926    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6962731d-8a32-42c9-8cac-e137bef90b68-secret-metrics-server-tls\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.446487 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.446458    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2d66\" (UniqueName: \"kubernetes.io/projected/6962731d-8a32-42c9-8cac-e137bef90b68-kube-api-access-k2d66\") pod \"metrics-server-66568f976d-rkd2f\" (UID: \"6962731d-8a32-42c9-8cac-e137bef90b68\") " pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.573773 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.573667    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:52:53.871015 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.870972    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66568f976d-rkd2f"]
Sep 16 15:52:53.992080 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:53.991769    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:52:54.400961 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:54.400926    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6962731d_8a32_42c9_8cac_e137bef90b68.slice/crio-d5dc97c58b797b7bab35377183f9e55f796736ed9b6763f0fb9db646c7d1387b WatchSource:0}: Error finding container d5dc97c58b797b7bab35377183f9e55f796736ed9b6763f0fb9db646c7d1387b: Status 404 returned error can't find the container with id d5dc97c58b797b7bab35377183f9e55f796736ed9b6763f0fb9db646c7d1387b
Sep 16 15:52:54.644805 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:54.644741    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66568f976d-rkd2f" event={"ID":"6962731d-8a32-42c9-8cac-e137bef90b68","Type":"ContainerStarted","Data":"d5dc97c58b797b7bab35377183f9e55f796736ed9b6763f0fb9db646c7d1387b"}
Sep 16 15:52:55.117578 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.117538    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 15:52:55.126451 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.126423    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.129502 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.129476    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 15:52:55.131228 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.131203    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 15:52:55.131559 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.131506    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 15:52:55.131679 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.131661    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 15:52:55.131797 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.131754    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 15:52:55.131907 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.131622    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 15:52:55.131971 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.131566    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 15:52:55.132761 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.132714    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 15:52:55.132922 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.132863    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 15:52:55.133204 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.133183    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 15:52:55.133303 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.133259    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 15:52:55.133455 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.133407    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f6eafte76m100\""
Sep 16 15:52:55.133783 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.133419    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 15:52:55.133783 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.133713    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 15:52:55.135161 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.135128    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4pnsn\""
Sep 16 15:52:55.136706 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.136637    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 15:52:55.139928 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.139905    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 15:52:55.154803 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.154779    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6swq\" (UniqueName: \"kubernetes.io/projected/326c729f-9a4e-4cfb-950c-30271d18a0ff-kube-api-access-z6swq\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.154973 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.154956    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155052 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155030    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155122 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155066    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155179 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155134    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155229 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155182    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/326c729f-9a4e-4cfb-950c-30271d18a0ff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155284 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155206    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155284 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155268    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155392 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155335    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155447 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155409    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/326c729f-9a4e-4cfb-950c-30271d18a0ff-config-out\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155506 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155448    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155602 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155472    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155602 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155584    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155739 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155700    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-config\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155881 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155852    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155942 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155894    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.155942 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-web-config\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.156043 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155965    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.156043 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.155992    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.156135 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.156023    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.256981 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.256741    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.256981 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.256792    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/326c729f-9a4e-4cfb-950c-30271d18a0ff-config-out\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.256981 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.256819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.256981 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.256845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.256981 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.256875    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.256981 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.256936    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-config\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.256981 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.256965    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.257416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.256990    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.257416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.257020    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-web-config\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.257416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.257045    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.257416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.257069    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.257416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.257100    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.257416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.257140    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6swq\" (UniqueName: \"kubernetes.io/projected/326c729f-9a4e-4cfb-950c-30271d18a0ff-kube-api-access-z6swq\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.257416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.257169    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.257416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.257202    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.257416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.257233    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.257416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.257269    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.257416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.257301    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/326c729f-9a4e-4cfb-950c-30271d18a0ff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.257416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.257329    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.257416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.257330    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.257416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.257355    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.258390 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.258361    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.261569 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.261362    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.261569 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.261410    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.262852 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.261809    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.262852 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.261941    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.262852 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.262168    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/326c729f-9a4e-4cfb-950c-30271d18a0ff-config-out\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.262852 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.262477    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.262852 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.262720    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.264196 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.263400    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.264196 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.263586    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-web-config\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.264196 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.263720    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.264196 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.263751    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.264196 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.264124    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.265890 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.265459    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.265890 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.265850    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/326c729f-9a4e-4cfb-950c-30271d18a0ff-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.266179 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.266146    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.266425 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.266345    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.269470 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.269390    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-config\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.272328 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.272287    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6swq\" (UniqueName: \"kubernetes.io/projected/326c729f-9a4e-4cfb-950c-30271d18a0ff-kube-api-access-z6swq\") pod \"prometheus-k8s-0\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.442225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.442146    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:52:55.654821 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.654780    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerStarted","Data":"8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c"}
Sep 16 15:52:55.655272 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.654882    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerStarted","Data":"2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31"}
Sep 16 15:52:55.655272 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.654926    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerStarted","Data":"04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98"}
Sep 16 15:52:55.655272 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.654968    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerStarted","Data":"9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0"}
Sep 16 15:52:55.655272 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.655007    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerStarted","Data":"eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103"}
Sep 16 15:52:55.659137 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.659008    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l" event={"ID":"2781fc36-78d3-4e86-b25c-6e99a11328fb","Type":"ContainerStarted","Data":"8a204b885e14a0fb8c4ac7f11122e35298bbdb7564dd842f63cc0c78d9d11654"}
Sep 16 15:52:55.659137 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.659090    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l" event={"ID":"2781fc36-78d3-4e86-b25c-6e99a11328fb","Type":"ContainerStarted","Data":"089faeda3609a4f1837b548e334d3458c48be1f414ad2a4bcf2ff07164cf5f94"}
Sep 16 15:52:55.659137 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.659136    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l" event={"ID":"2781fc36-78d3-4e86-b25c-6e99a11328fb","Type":"ContainerStarted","Data":"f8e9d0839e66dfe6fdbd852055bb07715329464653fe1c653c4b5677dfb66b25"}
Sep 16 15:52:55.729511 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:55.729361    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 15:52:56.015441 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:52:56.015340    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326c729f_9a4e_4cfb_950c_30271d18a0ff.slice/crio-efcd130deff226d467fa9fa8faebfedf02e2b367a8877d38a937330fc9ffa47c WatchSource:0}: Error finding container efcd130deff226d467fa9fa8faebfedf02e2b367a8877d38a937330fc9ffa47c: Status 404 returned error can't find the container with id efcd130deff226d467fa9fa8faebfedf02e2b367a8877d38a937330fc9ffa47c
Sep 16 15:52:56.670970 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:56.670932    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l" event={"ID":"2781fc36-78d3-4e86-b25c-6e99a11328fb","Type":"ContainerStarted","Data":"495b40d7739ea23f30bcc5f6570de51d088b9960d9854114d870b8bc280d15a5"}
Sep 16 15:52:56.671453 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:56.671014    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l" event={"ID":"2781fc36-78d3-4e86-b25c-6e99a11328fb","Type":"ContainerStarted","Data":"2fad82c791bf11f1e4477ff48f5c88ef0b9b9eb94cb778dd6bc9d62a14034f9c"}
Sep 16 15:52:56.671453 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:56.671042    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l" event={"ID":"2781fc36-78d3-4e86-b25c-6e99a11328fb","Type":"ContainerStarted","Data":"58ed39325192242a99cb23cc043140e1101e28eec5cbdb5c306f0e843c8d6bf3"}
Sep 16 15:52:56.671453 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:56.671070    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:52:56.672861 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:56.672831    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66568f976d-rkd2f" event={"ID":"6962731d-8a32-42c9-8cac-e137bef90b68","Type":"ContainerStarted","Data":"8900e127b73d498ddeb9502359bff19ec2683faf3e2655589d40ce16bfa1383a"}
Sep 16 15:52:56.674304 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:56.674282    2574 generic.go:356] "Generic (PLEG): container finished" podID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerID="2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7" exitCode=0
Sep 16 15:52:56.674384 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:56.674363    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerDied","Data":"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"}
Sep 16 15:52:56.674443 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:56.674429    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerStarted","Data":"efcd130deff226d467fa9fa8faebfedf02e2b367a8877d38a937330fc9ffa47c"}
Sep 16 15:52:56.678086 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:56.678061    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerStarted","Data":"965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a"}
Sep 16 15:52:56.691931 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:56.691862    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l" podStartSLOduration=2.318303398 podStartE2EDuration="6.691842336s" totalImagesPullingTime="4.373538938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:52:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:56.690579072 +0000 UTC m=+173.210201851" watchObservedRunningTime="2026-09-16 15:52:56.691842336 +0000 UTC m=+173.211465114"
Sep 16 15:52:56.717100 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:56.717052    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.625166057 podStartE2EDuration="6.717036325s" totalImagesPullingTime="4.075556739s" totalInitContainerRuntime="16.313529ms" isStatefulPod=false podCreationTimestamp="2026-09-16 15:52:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:56.715164772 +0000 UTC m=+173.234787586" watchObservedRunningTime="2026-09-16 15:52:56.717036325 +0000 UTC m=+173.236659101"
Sep 16 15:52:56.762630 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:56.762281    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-66568f976d-rkd2f" podStartSLOduration=1.95901897 podStartE2EDuration="3.762259669s" totalImagesPullingTime="1.803240699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:52:56.76206649 +0000 UTC m=+173.281689268" watchObservedRunningTime="2026-09-16 15:52:56.762259669 +0000 UTC m=+173.281882449"
Sep 16 15:52:59.613323 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:52:59.613286    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wq8bz"
Sep 16 15:53:00.702673 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:00.702630    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerStarted","Data":"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"}
Sep 16 15:53:00.703139 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:00.702726    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerStarted","Data":"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"}
Sep 16 15:53:00.703139 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:00.702769    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerStarted","Data":"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"}
Sep 16 15:53:00.703139 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:00.702809    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerStarted","Data":"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"}
Sep 16 15:53:00.703139 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:00.702846    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerStarted","Data":"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"}
Sep 16 15:53:00.703139 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:00.702879    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerStarted","Data":"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"}
Sep 16 15:53:00.729678 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:00.729615    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.532641057 podStartE2EDuration="5.729593788s" totalImagesPullingTime="3.168230729s" totalInitContainerRuntime="28.722002ms" isStatefulPod=false podCreationTimestamp="2026-09-16 15:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:53:00.726867299 +0000 UTC m=+177.246490076" watchObservedRunningTime="2026-09-16 15:53:00.729593788 +0000 UTC m=+177.249216566"
Sep 16 15:53:02.689939 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:02.689910    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57cb8f89c6-7h58l"
Sep 16 15:53:05.442813 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:05.442772    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:53:11.762852 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:11.762809    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-979kp" event={"ID":"15e1a31e-0ac4-4e41-9d1a-be65065d6089","Type":"ContainerStarted","Data":"95afb1a74a00436e0fa632abbec297e138e41fbbde441559a2b3d474b05676d7"}
Sep 16 15:53:11.763359 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:11.763161    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-979kp"
Sep 16 15:53:11.780983 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:11.780925    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-979kp" podStartSLOduration=2.560175532 podStartE2EDuration="21.780903563s" totalImagesPullingTime="19.220728031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:53:11.779252553 +0000 UTC m=+188.298875359" watchObservedRunningTime="2026-09-16 15:53:11.780903563 +0000 UTC m=+188.300526341"
Sep 16 15:53:11.781420 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:11.781393    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-979kp"
Sep 16 15:53:13.574048 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:13.574009    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:53:13.574492 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:13.574100    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:53:20.804381 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:20.804345    2574 generic.go:356] "Generic (PLEG): container finished" podID="29cf843e-24ce-4d87-baea-993e52f3d2e4" containerID="0d7cda8c12bb7c1d51085b9390bbf420a0a1163daf5f941822b335398ac0d1a8" exitCode=0
Sep 16 15:53:20.804822 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:20.804421    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-whb7g" event={"ID":"29cf843e-24ce-4d87-baea-993e52f3d2e4","Type":"ContainerDied","Data":"0d7cda8c12bb7c1d51085b9390bbf420a0a1163daf5f941822b335398ac0d1a8"}
Sep 16 15:53:20.804822 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:20.804765    2574 scope.go:117] "RemoveContainer" containerID="0d7cda8c12bb7c1d51085b9390bbf420a0a1163daf5f941822b335398ac0d1a8"
Sep 16 15:53:21.810705 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:21.810669    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-whb7g" event={"ID":"29cf843e-24ce-4d87-baea-993e52f3d2e4","Type":"ContainerStarted","Data":"0d82396601ef5a1fcb74ec5879c74395d920bc3c86a6750419968d47c342eaa7"}
Sep 16 15:53:33.583276 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:33.583247    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:53:33.587537 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:33.587490    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-66568f976d-rkd2f"
Sep 16 15:53:40.896584 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:40.896549    2574 generic.go:356] "Generic (PLEG): container finished" podID="d89c5398-15a5-4797-9759-63216417b9a4" containerID="e1213fe024128987d04e5d7304b9acf3db3d8274b71565f7e6cf72d3bd7d8542" exitCode=0
Sep 16 15:53:40.897124 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:40.896614    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44" event={"ID":"d89c5398-15a5-4797-9759-63216417b9a4","Type":"ContainerDied","Data":"e1213fe024128987d04e5d7304b9acf3db3d8274b71565f7e6cf72d3bd7d8542"}
Sep 16 15:53:40.897124 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:40.897017    2574 scope.go:117] "RemoveContainer" containerID="e1213fe024128987d04e5d7304b9acf3db3d8274b71565f7e6cf72d3bd7d8542"
Sep 16 15:53:41.902212 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:41.902179    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-65p44" event={"ID":"d89c5398-15a5-4797-9759-63216417b9a4","Type":"ContainerStarted","Data":"b19ab34711f28a551a2657aa06d19b73e5bb869aa1ee2447b9fcaff502c77892"}
Sep 16 15:53:55.442773 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:55.442733    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:53:55.458911 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:55.458880    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:53:55.971029 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:53:55.970996    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:09.191828 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:09.191798    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 15:54:09.192293 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:09.192091    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="alertmanager" containerID="cri-o://eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103" gracePeriod=120
Sep 16 15:54:09.192293 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:09.192173    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="kube-rbac-proxy-metric" containerID="cri-o://8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c" gracePeriod=120
Sep 16 15:54:09.192293 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:09.192221    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="prom-label-proxy" containerID="cri-o://965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a" gracePeriod=120
Sep 16 15:54:09.192293 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:09.192212    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="kube-rbac-proxy" containerID="cri-o://2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31" gracePeriod=120
Sep 16 15:54:09.192293 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:09.192183    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="config-reloader" containerID="cri-o://9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0" gracePeriod=120
Sep 16 15:54:09.192566 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:09.192177    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="kube-rbac-proxy-web" containerID="cri-o://04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98" gracePeriod=120
Sep 16 15:54:10.014274 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.014242    2574 generic.go:356] "Generic (PLEG): container finished" podID="916482a0-583c-40c2-96a9-f97a21c8652f" containerID="965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a" exitCode=0
Sep 16 15:54:10.014274 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.014265    2574 generic.go:356] "Generic (PLEG): container finished" podID="916482a0-583c-40c2-96a9-f97a21c8652f" containerID="2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31" exitCode=0
Sep 16 15:54:10.014274 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.014271    2574 generic.go:356] "Generic (PLEG): container finished" podID="916482a0-583c-40c2-96a9-f97a21c8652f" containerID="9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0" exitCode=0
Sep 16 15:54:10.014274 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.014276    2574 generic.go:356] "Generic (PLEG): container finished" podID="916482a0-583c-40c2-96a9-f97a21c8652f" containerID="eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103" exitCode=0
Sep 16 15:54:10.014546 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.014294    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerDied","Data":"965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a"}
Sep 16 15:54:10.014546 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.014342    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerDied","Data":"2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31"}
Sep 16 15:54:10.014546 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.014370    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerDied","Data":"9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0"}
Sep 16 15:54:10.014546 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.014392    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerDied","Data":"eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103"}
Sep 16 15:54:10.457775 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.457749    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:10.565214 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.565136    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/916482a0-583c-40c2-96a9-f97a21c8652f-tls-assets\") pod \"916482a0-583c-40c2-96a9-f97a21c8652f\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") "
Sep 16 15:54:10.565214 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.565185    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"916482a0-583c-40c2-96a9-f97a21c8652f\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") "
Sep 16 15:54:10.565214 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.565213    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-web-config\") pod \"916482a0-583c-40c2-96a9-f97a21c8652f\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") "
Sep 16 15:54:10.565474 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.565268    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-metrics-client-ca\") pod \"916482a0-583c-40c2-96a9-f97a21c8652f\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") "
Sep 16 15:54:10.565474 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.565302    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gsjt\" (UniqueName: \"kubernetes.io/projected/916482a0-583c-40c2-96a9-f97a21c8652f-kube-api-access-2gsjt\") pod \"916482a0-583c-40c2-96a9-f97a21c8652f\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") "
Sep 16 15:54:10.565474 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.565421    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-config-volume\") pod \"916482a0-583c-40c2-96a9-f97a21c8652f\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") "
Sep 16 15:54:10.565642 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.565481    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy\") pod \"916482a0-583c-40c2-96a9-f97a21c8652f\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") "
Sep 16 15:54:10.565642 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.565545    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-main-db\") pod \"916482a0-583c-40c2-96a9-f97a21c8652f\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") "
Sep 16 15:54:10.565642 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.565583    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-main-tls\") pod \"916482a0-583c-40c2-96a9-f97a21c8652f\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") "
Sep 16 15:54:10.565642 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.565615    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy-web\") pod \"916482a0-583c-40c2-96a9-f97a21c8652f\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") "
Sep 16 15:54:10.565838 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.565674    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-cluster-tls-config\") pod \"916482a0-583c-40c2-96a9-f97a21c8652f\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") "
Sep 16 15:54:10.565838 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.565697    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "916482a0-583c-40c2-96a9-f97a21c8652f" (UID: "916482a0-583c-40c2-96a9-f97a21c8652f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:54:10.565838 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.565718    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/916482a0-583c-40c2-96a9-f97a21c8652f-config-out\") pod \"916482a0-583c-40c2-96a9-f97a21c8652f\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") "
Sep 16 15:54:10.565838 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.565758    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-trusted-ca-bundle\") pod \"916482a0-583c-40c2-96a9-f97a21c8652f\" (UID: \"916482a0-583c-40c2-96a9-f97a21c8652f\") "
Sep 16 15:54:10.566118 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.566044    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-metrics-client-ca\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:10.566495 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.566465    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "916482a0-583c-40c2-96a9-f97a21c8652f" (UID: "916482a0-583c-40c2-96a9-f97a21c8652f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:54:10.566897 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.566868    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "916482a0-583c-40c2-96a9-f97a21c8652f" (UID: "916482a0-583c-40c2-96a9-f97a21c8652f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:54:10.568346 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.567919    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916482a0-583c-40c2-96a9-f97a21c8652f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "916482a0-583c-40c2-96a9-f97a21c8652f" (UID: "916482a0-583c-40c2-96a9-f97a21c8652f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:54:10.568959 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.568932    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "916482a0-583c-40c2-96a9-f97a21c8652f" (UID: "916482a0-583c-40c2-96a9-f97a21c8652f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:10.569057 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.569001    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916482a0-583c-40c2-96a9-f97a21c8652f-kube-api-access-2gsjt" (OuterVolumeSpecName: "kube-api-access-2gsjt") pod "916482a0-583c-40c2-96a9-f97a21c8652f" (UID: "916482a0-583c-40c2-96a9-f97a21c8652f"). InnerVolumeSpecName "kube-api-access-2gsjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:54:10.569057 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.569006    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916482a0-583c-40c2-96a9-f97a21c8652f-config-out" (OuterVolumeSpecName: "config-out") pod "916482a0-583c-40c2-96a9-f97a21c8652f" (UID: "916482a0-583c-40c2-96a9-f97a21c8652f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:54:10.569249 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.569224    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "916482a0-583c-40c2-96a9-f97a21c8652f" (UID: "916482a0-583c-40c2-96a9-f97a21c8652f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:10.569533 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.569498    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "916482a0-583c-40c2-96a9-f97a21c8652f" (UID: "916482a0-583c-40c2-96a9-f97a21c8652f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:10.570376 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.570351    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-config-volume" (OuterVolumeSpecName: "config-volume") pod "916482a0-583c-40c2-96a9-f97a21c8652f" (UID: "916482a0-583c-40c2-96a9-f97a21c8652f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:10.570608 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.570585    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "916482a0-583c-40c2-96a9-f97a21c8652f" (UID: "916482a0-583c-40c2-96a9-f97a21c8652f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:10.573384 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.573289    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "916482a0-583c-40c2-96a9-f97a21c8652f" (UID: "916482a0-583c-40c2-96a9-f97a21c8652f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:10.579403 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.579380    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-web-config" (OuterVolumeSpecName: "web-config") pod "916482a0-583c-40c2-96a9-f97a21c8652f" (UID: "916482a0-583c-40c2-96a9-f97a21c8652f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:10.666447 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.666413    2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-main-db\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:10.666447 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.666442    2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-main-tls\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:10.666447 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.666451    2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:10.666679 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.666462    2574 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-cluster-tls-config\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:10.666679 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.666470    2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/916482a0-583c-40c2-96a9-f97a21c8652f-config-out\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:10.666679 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.666479    2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/916482a0-583c-40c2-96a9-f97a21c8652f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:10.666679 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.666487    2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/916482a0-583c-40c2-96a9-f97a21c8652f-tls-assets\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:10.666679 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.666495    2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:10.666679 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.666504    2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-web-config\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:10.666679 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.666533    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gsjt\" (UniqueName: \"kubernetes.io/projected/916482a0-583c-40c2-96a9-f97a21c8652f-kube-api-access-2gsjt\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:10.666679 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.666542    2574 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-config-volume\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:10.666679 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:10.666552    2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/916482a0-583c-40c2-96a9-f97a21c8652f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:11.021279 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.021243    2574 generic.go:356] "Generic (PLEG): container finished" podID="916482a0-583c-40c2-96a9-f97a21c8652f" containerID="8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c" exitCode=0
Sep 16 15:54:11.021279 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.021268    2574 generic.go:356] "Generic (PLEG): container finished" podID="916482a0-583c-40c2-96a9-f97a21c8652f" containerID="04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98" exitCode=0
Sep 16 15:54:11.021539 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.021323    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.021539 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.021321    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerDied","Data":"8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c"}
Sep 16 15:54:11.021539 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.021405    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerDied","Data":"04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98"}
Sep 16 15:54:11.021539 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.021449    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"916482a0-583c-40c2-96a9-f97a21c8652f","Type":"ContainerDied","Data":"48593ea17fcde2bc0967500ee224be70bdbf3090fb85a384f8e84726469c3382"}
Sep 16 15:54:11.021539 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.021467    2574 scope.go:117] "RemoveContainer" containerID="965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a"
Sep 16 15:54:11.029732 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.029709    2574 scope.go:117] "RemoveContainer" containerID="8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c"
Sep 16 15:54:11.036313 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.036293    2574 scope.go:117] "RemoveContainer" containerID="2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31"
Sep 16 15:54:11.043508 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.043487    2574 scope.go:117] "RemoveContainer" containerID="04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98"
Sep 16 15:54:11.043931 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.043915    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 15:54:11.049136 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.049112    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 15:54:11.051556 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.051532    2574 scope.go:117] "RemoveContainer" containerID="9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0"
Sep 16 15:54:11.058315 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.058295    2574 scope.go:117] "RemoveContainer" containerID="eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103"
Sep 16 15:54:11.065338 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.065300    2574 scope.go:117] "RemoveContainer" containerID="da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13"
Sep 16 15:54:11.072408 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.072386    2574 scope.go:117] "RemoveContainer" containerID="965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a"
Sep 16 15:54:11.072687 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:54:11.072665    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a\": container with ID starting with 965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a not found: ID does not exist" containerID="965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a"
Sep 16 15:54:11.072738 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.072700    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a"} err="failed to get container status \"965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a\": rpc error: code = NotFound desc = could not find container \"965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a\": container with ID starting with 965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a not found: ID does not exist"
Sep 16 15:54:11.072738 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.072733    2574 scope.go:117] "RemoveContainer" containerID="8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c"
Sep 16 15:54:11.072981 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:54:11.072965    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c\": container with ID starting with 8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c not found: ID does not exist" containerID="8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c"
Sep 16 15:54:11.073036 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.072987    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c"} err="failed to get container status \"8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c\": rpc error: code = NotFound desc = could not find container \"8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c\": container with ID starting with 8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c not found: ID does not exist"
Sep 16 15:54:11.073036 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.073002    2574 scope.go:117] "RemoveContainer" containerID="2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31"
Sep 16 15:54:11.073201 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:54:11.073187    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31\": container with ID starting with 2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31 not found: ID does not exist" containerID="2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31"
Sep 16 15:54:11.073238 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.073203    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31"} err="failed to get container status \"2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31\": rpc error: code = NotFound desc = could not find container \"2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31\": container with ID starting with 2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31 not found: ID does not exist"
Sep 16 15:54:11.073238 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.073216    2574 scope.go:117] "RemoveContainer" containerID="04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98"
Sep 16 15:54:11.073452 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:54:11.073437    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98\": container with ID starting with 04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98 not found: ID does not exist" containerID="04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98"
Sep 16 15:54:11.073493 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.073459    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98"} err="failed to get container status \"04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98\": rpc error: code = NotFound desc = could not find container \"04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98\": container with ID starting with 04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98 not found: ID does not exist"
Sep 16 15:54:11.073493 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.073471    2574 scope.go:117] "RemoveContainer" containerID="9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0"
Sep 16 15:54:11.073719 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:54:11.073693    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0\": container with ID starting with 9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0 not found: ID does not exist" containerID="9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0"
Sep 16 15:54:11.073719 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.073713    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0"} err="failed to get container status \"9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0\": rpc error: code = NotFound desc = could not find container \"9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0\": container with ID starting with 9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0 not found: ID does not exist"
Sep 16 15:54:11.073813 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.073725    2574 scope.go:117] "RemoveContainer" containerID="eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103"
Sep 16 15:54:11.073952 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:54:11.073932    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103\": container with ID starting with eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103 not found: ID does not exist" containerID="eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103"
Sep 16 15:54:11.074027 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.073953    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103"} err="failed to get container status \"eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103\": rpc error: code = NotFound desc = could not find container \"eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103\": container with ID starting with eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103 not found: ID does not exist"
Sep 16 15:54:11.074027 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.073968    2574 scope.go:117] "RemoveContainer" containerID="da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13"
Sep 16 15:54:11.074246 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:54:11.074222    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13\": container with ID starting with da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13 not found: ID does not exist" containerID="da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13"
Sep 16 15:54:11.074293 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.074244    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13"} err="failed to get container status \"da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13\": rpc error: code = NotFound desc = could not find container \"da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13\": container with ID starting with da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13 not found: ID does not exist"
Sep 16 15:54:11.074293 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.074260    2574 scope.go:117] "RemoveContainer" containerID="965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a"
Sep 16 15:54:11.074493 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.074477    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a"} err="failed to get container status \"965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a\": rpc error: code = NotFound desc = could not find container \"965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a\": container with ID starting with 965283bfb5e3e46cd3d4fe74571e641393b4d881d957bccb1ba9ef80050f6c5a not found: ID does not exist"
Sep 16 15:54:11.074543 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.074493    2574 scope.go:117] "RemoveContainer" containerID="8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c"
Sep 16 15:54:11.074731 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.074716    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c"} err="failed to get container status \"8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c\": rpc error: code = NotFound desc = could not find container \"8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c\": container with ID starting with 8b012be4235e650fcfe12a749c06553cbecaaa80cb939dba16a7b7ad8db6010c not found: ID does not exist"
Sep 16 15:54:11.074793 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.074731    2574 scope.go:117] "RemoveContainer" containerID="2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31"
Sep 16 15:54:11.074963 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.074946    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31"} err="failed to get container status \"2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31\": rpc error: code = NotFound desc = could not find container \"2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31\": container with ID starting with 2cabdf7b8e2ebf7156a046cf86ea96c47ec6f91c79c323806b714ca373d97b31 not found: ID does not exist"
Sep 16 15:54:11.074963 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.074962    2574 scope.go:117] "RemoveContainer" containerID="04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98"
Sep 16 15:54:11.075206 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.075189    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98"} err="failed to get container status \"04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98\": rpc error: code = NotFound desc = could not find container \"04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98\": container with ID starting with 04783cb67bc6986446b639da9a5eb727821f74e491f122d05ba01e414cbc3e98 not found: ID does not exist"
Sep 16 15:54:11.075251 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.075206    2574 scope.go:117] "RemoveContainer" containerID="9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0"
Sep 16 15:54:11.075409 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.075394    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0"} err="failed to get container status \"9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0\": rpc error: code = NotFound desc = could not find container \"9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0\": container with ID starting with 9b9e4f333fda5f9bf6c7eaec7e83367ca6a2a1d3c3fddb1b941dfe3d41f9d0f0 not found: ID does not exist"
Sep 16 15:54:11.075451 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.075410    2574 scope.go:117] "RemoveContainer" containerID="eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103"
Sep 16 15:54:11.075625 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.075610    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103"} err="failed to get container status \"eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103\": rpc error: code = NotFound desc = could not find container \"eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103\": container with ID starting with eeb61951699e8ab0f4041e805a85e06da11e2ff12ad475c7b8ba1a413feb6103 not found: ID does not exist"
Sep 16 15:54:11.075625 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.075624    2574 scope.go:117] "RemoveContainer" containerID="da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13"
Sep 16 15:54:11.075857 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.075838    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13"} err="failed to get container status \"da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13\": rpc error: code = NotFound desc = could not find container \"da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13\": container with ID starting with da4e7ba83a3d279306d44f013fa7f7542c41ae0a86d34b04ae34aad9e7b33a13 not found: ID does not exist"
Sep 16 15:54:11.079354 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079331    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 15:54:11.079718 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079702    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="alertmanager"
Sep 16 15:54:11.079801 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079720    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="alertmanager"
Sep 16 15:54:11.079801 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079738    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="init-config-reloader"
Sep 16 15:54:11.079801 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079747    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="init-config-reloader"
Sep 16 15:54:11.079801 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079757    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="kube-rbac-proxy"
Sep 16 15:54:11.079801 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079765    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="kube-rbac-proxy"
Sep 16 15:54:11.079801 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079780    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="prom-label-proxy"
Sep 16 15:54:11.079801 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079788    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="prom-label-proxy"
Sep 16 15:54:11.080109 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079808    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="config-reloader"
Sep 16 15:54:11.080109 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079817    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="config-reloader"
Sep 16 15:54:11.080109 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079829    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="kube-rbac-proxy-web"
Sep 16 15:54:11.080109 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079837    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="kube-rbac-proxy-web"
Sep 16 15:54:11.080109 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079845    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="kube-rbac-proxy-metric"
Sep 16 15:54:11.080109 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079853    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="kube-rbac-proxy-metric"
Sep 16 15:54:11.080109 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079942    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="config-reloader"
Sep 16 15:54:11.080109 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079953    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="alertmanager"
Sep 16 15:54:11.080109 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079963    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="kube-rbac-proxy-metric"
Sep 16 15:54:11.080109 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079973    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="prom-label-proxy"
Sep 16 15:54:11.080109 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079982    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="kube-rbac-proxy"
Sep 16 15:54:11.080109 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.079989    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" containerName="kube-rbac-proxy-web"
Sep 16 15:54:11.086817 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.086794    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.089062 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.089038    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lvf5g\""
Sep 16 15:54:11.089148 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.089065    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 15:54:11.089148 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.089112    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 15:54:11.089301 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.089284    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 15:54:11.089399 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.089300    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 15:54:11.089399 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.089330    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 15:54:11.089399 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.089374    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 15:54:11.089641 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.089627    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 15:54:11.089719 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.089702    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 15:54:11.095307 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.095266    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 15:54:11.096418 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.096396    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 15:54:11.271620 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.271552    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.271620 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.271609    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46gsw\" (UniqueName: \"kubernetes.io/projected/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-kube-api-access-46gsw\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.271752 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.271684    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.271752 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.271714    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.271818 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.271758    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-config-volume\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.271818 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.271801    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.271875 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.271827    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-config-out\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.271907 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.271880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.271907 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.271897    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.271970 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.271922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.271970 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.271946    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.272032 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.271970    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.272032 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.271991    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-web-config\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.373368 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.373334    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.373368 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.373370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.373618 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.373405    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.373618 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.373436    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-web-config\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.373618 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.373469    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.373618 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.373551    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46gsw\" (UniqueName: \"kubernetes.io/projected/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-kube-api-access-46gsw\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.373618 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.373585    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.373618 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.373608    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.373878 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.373649    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-config-volume\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.373878 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.373683    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.373878 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.373709    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-config-out\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.373878 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.373740    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.373878 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.373763    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.374963 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.374787    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.375184 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.375161    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.375286 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.375161    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.376532 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.376492    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.376628 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.376486    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.376686 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.376653    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-web-config\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.377104 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.377079    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.377186 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.377136    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.377297 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.377274    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.377692 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.377676    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-config-out\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.377989 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.377974    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.378282 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.378265    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-config-volume\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.381470 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.381447    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46gsw\" (UniqueName: \"kubernetes.io/projected/1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23-kube-api-access-46gsw\") pod \"alertmanager-main-0\" (UID: \"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.397461 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.397442    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 15:54:11.675717 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.675681    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 15:54:11.679011 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:54:11.678986    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1718f3eb_dd98_40a9_bab4_d3b1f4fb0a23.slice/crio-8cb69473dd3fa221093c38859c585316fe649941c83bbe2e4adb0cb78c851a90 WatchSource:0}: Error finding container 8cb69473dd3fa221093c38859c585316fe649941c83bbe2e4adb0cb78c851a90: Status 404 returned error can't find the container with id 8cb69473dd3fa221093c38859c585316fe649941c83bbe2e4adb0cb78c851a90
Sep 16 15:54:11.993247 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:11.993173    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="916482a0-583c-40c2-96a9-f97a21c8652f" path="/var/lib/kubelet/pods/916482a0-583c-40c2-96a9-f97a21c8652f/volumes"
Sep 16 15:54:12.026692 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:12.026662    2574 generic.go:356] "Generic (PLEG): container finished" podID="1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23" containerID="c10a4b68ac3a3d1c4c121efdb8e530d0f62e49cdf8fa0cd054df1272009276e7" exitCode=0
Sep 16 15:54:12.026827 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:12.026753    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23","Type":"ContainerDied","Data":"c10a4b68ac3a3d1c4c121efdb8e530d0f62e49cdf8fa0cd054df1272009276e7"}
Sep 16 15:54:12.026827 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:12.026821    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23","Type":"ContainerStarted","Data":"8cb69473dd3fa221093c38859c585316fe649941c83bbe2e4adb0cb78c851a90"}
Sep 16 15:54:13.034581 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.034543    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23","Type":"ContainerStarted","Data":"ab127a4fa4050358538519e0f4e47c8334672c9311dfa5f6a3681e976a12988b"}
Sep 16 15:54:13.034932 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.034612    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23","Type":"ContainerStarted","Data":"f405c782df361af5321985ebd2e8a86ee444859e9bb1aad20fd992f48a15ed94"}
Sep 16 15:54:13.034932 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.034646    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23","Type":"ContainerStarted","Data":"0e992ae815c933eadd941cf772f42b0d48aa9efbe61fdb630f080e37a6a4c7b3"}
Sep 16 15:54:13.034932 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.034684    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23","Type":"ContainerStarted","Data":"7468ecb93f6fb83cee11819ca67351d16153227c469023c958d4c207518421ef"}
Sep 16 15:54:13.034932 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.034723    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23","Type":"ContainerStarted","Data":"c718bcc33a26f5cee959d041cd2a1444f3e9157e8a8f2408498dfb9bc443e0ca"}
Sep 16 15:54:13.034932 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.034758    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23","Type":"ContainerStarted","Data":"0be73a097c6a7d4235d39250c4529f9d541b135fc6a77d5460de3756f150bf09"}
Sep 16 15:54:13.061004 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.060947    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.041842803 podStartE2EDuration="2.060926008s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.083205ms" isStatefulPod=false podCreationTimestamp="2026-09-16 15:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:54:13.058708094 +0000 UTC m=+249.578331002" watchObservedRunningTime="2026-09-16 15:54:13.060926008 +0000 UTC m=+249.580548783"
Sep 16 15:54:13.207211 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.207178    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"]
Sep 16 15:54:13.210577 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.210557    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.212874 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.212855    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 15:54:13.212997 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.212895    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 15:54:13.212997 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.212895    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-pm5vw\""
Sep 16 15:54:13.212997 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.212905    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 15:54:13.213274 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.213258    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 15:54:13.213363 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.213331    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 15:54:13.223607 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.223583    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 15:54:13.227140 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.227121    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"]
Sep 16 15:54:13.289753 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.289673    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/86294e16-98ef-4a59-aae0-68600f10abbf-secret-telemeter-client\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.289753 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.289708    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/86294e16-98ef-4a59-aae0-68600f10abbf-telemeter-client-tls\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.289753 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.289725    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86294e16-98ef-4a59-aae0-68600f10abbf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.289947 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.289790    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86294e16-98ef-4a59-aae0-68600f10abbf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.289947 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.289811    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86294e16-98ef-4a59-aae0-68600f10abbf-serving-certs-ca-bundle\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.289947 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.289844    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86294e16-98ef-4a59-aae0-68600f10abbf-metrics-client-ca\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.289947 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.289867    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d69w\" (UniqueName: \"kubernetes.io/projected/86294e16-98ef-4a59-aae0-68600f10abbf-kube-api-access-2d69w\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.289947 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.289884    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/86294e16-98ef-4a59-aae0-68600f10abbf-federate-client-tls\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.390771 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.390731    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86294e16-98ef-4a59-aae0-68600f10abbf-serving-certs-ca-bundle\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.390914 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.390783    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86294e16-98ef-4a59-aae0-68600f10abbf-metrics-client-ca\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.390914 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.390818    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d69w\" (UniqueName: \"kubernetes.io/projected/86294e16-98ef-4a59-aae0-68600f10abbf-kube-api-access-2d69w\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.390914 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.390845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/86294e16-98ef-4a59-aae0-68600f10abbf-federate-client-tls\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.390914 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.390907    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/86294e16-98ef-4a59-aae0-68600f10abbf-secret-telemeter-client\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.391196 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.390933    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/86294e16-98ef-4a59-aae0-68600f10abbf-telemeter-client-tls\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.391196 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.390957    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86294e16-98ef-4a59-aae0-68600f10abbf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.391196 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.391040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86294e16-98ef-4a59-aae0-68600f10abbf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.391638 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.391602    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86294e16-98ef-4a59-aae0-68600f10abbf-serving-certs-ca-bundle\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.391811 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.391619    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86294e16-98ef-4a59-aae0-68600f10abbf-metrics-client-ca\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.391936 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.391845    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86294e16-98ef-4a59-aae0-68600f10abbf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.393536 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.393494    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/86294e16-98ef-4a59-aae0-68600f10abbf-secret-telemeter-client\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.393616 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.393547    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/86294e16-98ef-4a59-aae0-68600f10abbf-federate-client-tls\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.393746 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.393730    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86294e16-98ef-4a59-aae0-68600f10abbf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.393912 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.393890    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/86294e16-98ef-4a59-aae0-68600f10abbf-telemeter-client-tls\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.398130 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.398105    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d69w\" (UniqueName: \"kubernetes.io/projected/86294e16-98ef-4a59-aae0-68600f10abbf-kube-api-access-2d69w\") pod \"telemeter-client-6786f7c4c4-8srfn\" (UID: \"86294e16-98ef-4a59-aae0-68600f10abbf\") " pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.484989 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.484957    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 15:54:13.485288 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.485248    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="prometheus" containerID="cri-o://b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab" gracePeriod=600
Sep 16 15:54:13.485427 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.485273    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="config-reloader" containerID="cri-o://b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7" gracePeriod=600
Sep 16 15:54:13.485427 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.485288    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="kube-rbac-proxy" containerID="cri-o://7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8" gracePeriod=600
Sep 16 15:54:13.485427 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.485272    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="kube-rbac-proxy-web" containerID="cri-o://70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7" gracePeriod=600
Sep 16 15:54:13.485427 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.485361    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce" gracePeriod=600
Sep 16 15:54:13.485427 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.485287    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="thanos-sidecar" containerID="cri-o://799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888" gracePeriod=600
Sep 16 15:54:13.525849 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.525824    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"
Sep 16 15:54:13.719332 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.719303    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:13.793491 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793461    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-tls\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.793632 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793496    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-kubelet-serving-ca-bundle\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.793632 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793548    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/326c729f-9a4e-4cfb-950c-30271d18a0ff-config-out\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.793632 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793568    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-web-config\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.793632 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793582    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-serving-certs-ca-bundle\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.793632 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793622    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-config\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.793872 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793643    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-metrics-client-ca\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.793872 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793672    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-2\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.793872 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793707    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/326c729f-9a4e-4cfb-950c-30271d18a0ff-tls-assets\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.793872 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793732    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-db\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.793872 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793758    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-kube-rbac-proxy\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.793872 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793781    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-0\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.793872 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793821    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6swq\" (UniqueName: \"kubernetes.io/projected/326c729f-9a4e-4cfb-950c-30271d18a0ff-kube-api-access-z6swq\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.793872 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793847    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.793872 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793872    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-1\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.794282 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793899    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-trusted-ca-bundle\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.794282 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793933    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.794282 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.793968    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-metrics-client-certs\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.794282 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.794001    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-grpc-tls\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.794282 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.794025    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-thanos-prometheus-http-client-file\") pod \"326c729f-9a4e-4cfb-950c-30271d18a0ff\" (UID: \"326c729f-9a4e-4cfb-950c-30271d18a0ff\") "
Sep 16 15:54:13.794282 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.794055    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:54:13.794592 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.794311    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.794592 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.794324    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:54:13.794689 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.794649    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:54:13.795396 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.795066    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:54:13.795396 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.795152    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:54:13.795862 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.795833    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:54:13.796781 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.796643    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326c729f-9a4e-4cfb-950c-30271d18a0ff-config-out" (OuterVolumeSpecName: "config-out") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:54:13.796781 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.796715    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:13.797267 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.796925    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:54:13.797801 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.797774    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:54:13.798189 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.798165    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:13.798416 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.798392    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326c729f-9a4e-4cfb-950c-30271d18a0ff-kube-api-access-z6swq" (OuterVolumeSpecName: "kube-api-access-z6swq") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "kube-api-access-z6swq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:54:13.799609 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.799571    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326c729f-9a4e-4cfb-950c-30271d18a0ff-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:54:13.799714 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.799677    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:13.799774 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.799723    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:13.799774 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.799753    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-config" (OuterVolumeSpecName: "config") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:13.800475 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.800444    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:13.800917 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.800881    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:13.801368 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.801339    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6786f7c4c4-8srfn"]
Sep 16 15:54:13.801368 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.801352    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:13.804158 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:54:13.804138    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86294e16_98ef_4a59_aae0_68600f10abbf.slice/crio-6463179de4b1164023fabf88c1fea16e07bc926a7e81cff762bdb97602011e3e WatchSource:0}: Error finding container 6463179de4b1164023fabf88c1fea16e07bc926a7e81cff762bdb97602011e3e: Status 404 returned error can't find the container with id 6463179de4b1164023fabf88c1fea16e07bc926a7e81cff762bdb97602011e3e
Sep 16 15:54:13.808373 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.808351    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-web-config" (OuterVolumeSpecName: "web-config") pod "326c729f-9a4e-4cfb-950c-30271d18a0ff" (UID: "326c729f-9a4e-4cfb-950c-30271d18a0ff"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:54:13.895099 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895071    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895099 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895094    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895105    2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/326c729f-9a4e-4cfb-950c-30271d18a0ff-config-out\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895114    2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-web-config\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895121    2574 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-config\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895128    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-configmap-metrics-client-ca\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895136    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895146    2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/326c729f-9a4e-4cfb-950c-30271d18a0ff-tls-assets\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895154    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-db\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895162    2574 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-kube-rbac-proxy\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895170    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895178    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6swq\" (UniqueName: \"kubernetes.io/projected/326c729f-9a4e-4cfb-950c-30271d18a0ff-kube-api-access-z6swq\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895186    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895194    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895202    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326c729f-9a4e-4cfb-950c-30271d18a0ff-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895213    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895221    2574 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-metrics-client-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895231    2574 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-secret-grpc-tls\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:13.895278 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:13.895238    2574 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/326c729f-9a4e-4cfb-950c-30271d18a0ff-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 15:54:14.039840 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.039805    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn" event={"ID":"86294e16-98ef-4a59-aae0-68600f10abbf","Type":"ContainerStarted","Data":"6463179de4b1164023fabf88c1fea16e07bc926a7e81cff762bdb97602011e3e"}
Sep 16 15:54:14.042951 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.042926    2574 generic.go:356] "Generic (PLEG): container finished" podID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerID="b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce" exitCode=0
Sep 16 15:54:14.043060 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.042954    2574 generic.go:356] "Generic (PLEG): container finished" podID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerID="7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8" exitCode=0
Sep 16 15:54:14.043060 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.042964    2574 generic.go:356] "Generic (PLEG): container finished" podID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerID="70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7" exitCode=0
Sep 16 15:54:14.043060 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.042972    2574 generic.go:356] "Generic (PLEG): container finished" podID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerID="799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888" exitCode=0
Sep 16 15:54:14.043060 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.042983    2574 generic.go:356] "Generic (PLEG): container finished" podID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerID="b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7" exitCode=0
Sep 16 15:54:14.043060 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.042991    2574 generic.go:356] "Generic (PLEG): container finished" podID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerID="b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab" exitCode=0
Sep 16 15:54:14.043060 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.043005    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerDied","Data":"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"}
Sep 16 15:54:14.043254 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.043015    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.043254 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.043083    2574 scope.go:117] "RemoveContainer" containerID="b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"
Sep 16 15:54:14.043254 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.043072    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerDied","Data":"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"}
Sep 16 15:54:14.043254 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.043193    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerDied","Data":"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"}
Sep 16 15:54:14.043254 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.043223    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerDied","Data":"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"}
Sep 16 15:54:14.043254 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.043253    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerDied","Data":"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"}
Sep 16 15:54:14.043511 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.043288    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerDied","Data":"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"}
Sep 16 15:54:14.043511 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.043312    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326c729f-9a4e-4cfb-950c-30271d18a0ff","Type":"ContainerDied","Data":"efcd130deff226d467fa9fa8faebfedf02e2b367a8877d38a937330fc9ffa47c"}
Sep 16 15:54:14.050810 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.050795    2574 scope.go:117] "RemoveContainer" containerID="7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"
Sep 16 15:54:14.057438 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.057421    2574 scope.go:117] "RemoveContainer" containerID="70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"
Sep 16 15:54:14.062410 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.062390    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 15:54:14.065022 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.064990    2574 scope.go:117] "RemoveContainer" containerID="799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"
Sep 16 15:54:14.066503 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.066484    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 15:54:14.073214 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.073196    2574 scope.go:117] "RemoveContainer" containerID="b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"
Sep 16 15:54:14.080043 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.080026    2574 scope.go:117] "RemoveContainer" containerID="b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"
Sep 16 15:54:14.086767 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.086751    2574 scope.go:117] "RemoveContainer" containerID="2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"
Sep 16 15:54:14.092990 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.092972    2574 scope.go:117] "RemoveContainer" containerID="b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"
Sep 16 15:54:14.094066 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:54:14.094036    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce\": container with ID starting with b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce not found: ID does not exist" containerID="b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"
Sep 16 15:54:14.094139 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.094068    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"} err="failed to get container status \"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce\": rpc error: code = NotFound desc = could not find container \"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce\": container with ID starting with b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce not found: ID does not exist"
Sep 16 15:54:14.094139 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.094087    2574 scope.go:117] "RemoveContainer" containerID="7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"
Sep 16 15:54:14.094336 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:54:14.094305    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8\": container with ID starting with 7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8 not found: ID does not exist" containerID="7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"
Sep 16 15:54:14.094400 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.094339    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"} err="failed to get container status \"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8\": rpc error: code = NotFound desc = could not find container \"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8\": container with ID starting with 7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8 not found: ID does not exist"
Sep 16 15:54:14.094400 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.094363    2574 scope.go:117] "RemoveContainer" containerID="70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"
Sep 16 15:54:14.094663 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:54:14.094640    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7\": container with ID starting with 70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7 not found: ID does not exist" containerID="70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"
Sep 16 15:54:14.094738 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.094672    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"} err="failed to get container status \"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7\": rpc error: code = NotFound desc = could not find container \"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7\": container with ID starting with 70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7 not found: ID does not exist"
Sep 16 15:54:14.094738 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.094693    2574 scope.go:117] "RemoveContainer" containerID="799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"
Sep 16 15:54:14.094876 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.094850    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 15:54:14.094960 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:54:14.094947    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888\": container with ID starting with 799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888 not found: ID does not exist" containerID="799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"
Sep 16 15:54:14.095005 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.094968    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"} err="failed to get container status \"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888\": rpc error: code = NotFound desc = could not find container \"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888\": container with ID starting with 799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888 not found: ID does not exist"
Sep 16 15:54:14.095005 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.094987    2574 scope.go:117] "RemoveContainer" containerID="b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"
Sep 16 15:54:14.095254 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095240    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="prometheus"
Sep 16 15:54:14.095254 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:54:14.095246    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7\": container with ID starting with b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7 not found: ID does not exist" containerID="b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"
Sep 16 15:54:14.095321 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095266    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"} err="failed to get container status \"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7\": rpc error: code = NotFound desc = could not find container \"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7\": container with ID starting with b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7 not found: ID does not exist"
Sep 16 15:54:14.095321 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095280    2574 scope.go:117] "RemoveContainer" containerID="b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"
Sep 16 15:54:14.095321 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095256    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="prometheus"
Sep 16 15:54:14.095424 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095336    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="thanos-sidecar"
Sep 16 15:54:14.095424 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095347    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="thanos-sidecar"
Sep 16 15:54:14.095424 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095359    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="kube-rbac-proxy-web"
Sep 16 15:54:14.095424 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095367    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="kube-rbac-proxy-web"
Sep 16 15:54:14.095424 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095376    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="kube-rbac-proxy"
Sep 16 15:54:14.095424 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095384    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="kube-rbac-proxy"
Sep 16 15:54:14.095424 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095395    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="kube-rbac-proxy-thanos"
Sep 16 15:54:14.095424 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095402    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="kube-rbac-proxy-thanos"
Sep 16 15:54:14.095424 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095423    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="config-reloader"
Sep 16 15:54:14.095750 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095431    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="config-reloader"
Sep 16 15:54:14.095750 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095446    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="init-config-reloader"
Sep 16 15:54:14.095750 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095455    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="init-config-reloader"
Sep 16 15:54:14.095750 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:54:14.095549    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab\": container with ID starting with b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab not found: ID does not exist" containerID="b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"
Sep 16 15:54:14.095750 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095569    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"} err="failed to get container status \"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab\": rpc error: code = NotFound desc = could not find container \"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab\": container with ID starting with b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab not found: ID does not exist"
Sep 16 15:54:14.095750 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095582    2574 scope.go:117] "RemoveContainer" containerID="2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"
Sep 16 15:54:14.095750 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095588    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="kube-rbac-proxy"
Sep 16 15:54:14.095750 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095603    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="config-reloader"
Sep 16 15:54:14.095750 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095612    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="prometheus"
Sep 16 15:54:14.095750 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095634    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="kube-rbac-proxy-web"
Sep 16 15:54:14.095750 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095646    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="kube-rbac-proxy-thanos"
Sep 16 15:54:14.095750 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095652    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" containerName="thanos-sidecar"
Sep 16 15:54:14.096147 ip-10-0-143-143 kubenswrapper[2574]: E0916 15:54:14.095829    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7\": container with ID starting with 2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7 not found: ID does not exist" containerID="2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"
Sep 16 15:54:14.096147 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095849    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"} err="failed to get container status \"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7\": rpc error: code = NotFound desc = could not find container \"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7\": container with ID starting with 2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7 not found: ID does not exist"
Sep 16 15:54:14.096147 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.095864    2574 scope.go:117] "RemoveContainer" containerID="b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"
Sep 16 15:54:14.096295 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.096174    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"} err="failed to get container status \"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce\": rpc error: code = NotFound desc = could not find container \"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce\": container with ID starting with b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce not found: ID does not exist"
Sep 16 15:54:14.096295 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.096202    2574 scope.go:117] "RemoveContainer" containerID="7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"
Sep 16 15:54:14.096756 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.096735    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"} err="failed to get container status \"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8\": rpc error: code = NotFound desc = could not find container \"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8\": container with ID starting with 7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8 not found: ID does not exist"
Sep 16 15:54:14.096756 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.096755    2574 scope.go:117] "RemoveContainer" containerID="70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"
Sep 16 15:54:14.096992 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.096973    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"} err="failed to get container status \"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7\": rpc error: code = NotFound desc = could not find container \"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7\": container with ID starting with 70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7 not found: ID does not exist"
Sep 16 15:54:14.097037 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.096993    2574 scope.go:117] "RemoveContainer" containerID="799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"
Sep 16 15:54:14.097225 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.097206    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"} err="failed to get container status \"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888\": rpc error: code = NotFound desc = could not find container \"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888\": container with ID starting with 799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888 not found: ID does not exist"
Sep 16 15:54:14.097271 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.097227    2574 scope.go:117] "RemoveContainer" containerID="b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"
Sep 16 15:54:14.097461 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.097441    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"} err="failed to get container status \"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7\": rpc error: code = NotFound desc = could not find container \"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7\": container with ID starting with b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7 not found: ID does not exist"
Sep 16 15:54:14.097513 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.097462    2574 scope.go:117] "RemoveContainer" containerID="b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"
Sep 16 15:54:14.097703 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.097689    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"} err="failed to get container status \"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab\": rpc error: code = NotFound desc = could not find container \"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab\": container with ID starting with b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab not found: ID does not exist"
Sep 16 15:54:14.097746 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.097703    2574 scope.go:117] "RemoveContainer" containerID="2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"
Sep 16 15:54:14.097892 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.097878    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"} err="failed to get container status \"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7\": rpc error: code = NotFound desc = could not find container \"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7\": container with ID starting with 2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7 not found: ID does not exist"
Sep 16 15:54:14.097932 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.097894    2574 scope.go:117] "RemoveContainer" containerID="b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"
Sep 16 15:54:14.098126 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.098076    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"} err="failed to get container status \"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce\": rpc error: code = NotFound desc = could not find container \"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce\": container with ID starting with b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce not found: ID does not exist"
Sep 16 15:54:14.098163 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.098128    2574 scope.go:117] "RemoveContainer" containerID="7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"
Sep 16 15:54:14.098334 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.098317    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"} err="failed to get container status \"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8\": rpc error: code = NotFound desc = could not find container \"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8\": container with ID starting with 7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8 not found: ID does not exist"
Sep 16 15:54:14.098376 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.098335    2574 scope.go:117] "RemoveContainer" containerID="70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"
Sep 16 15:54:14.098610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.098566    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"} err="failed to get container status \"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7\": rpc error: code = NotFound desc = could not find container \"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7\": container with ID starting with 70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7 not found: ID does not exist"
Sep 16 15:54:14.098610 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.098586    2574 scope.go:117] "RemoveContainer" containerID="799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"
Sep 16 15:54:14.098809 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.098792    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"} err="failed to get container status \"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888\": rpc error: code = NotFound desc = could not find container \"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888\": container with ID starting with 799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888 not found: ID does not exist"
Sep 16 15:54:14.098850 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.098809    2574 scope.go:117] "RemoveContainer" containerID="b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"
Sep 16 15:54:14.099031 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.099014    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"} err="failed to get container status \"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7\": rpc error: code = NotFound desc = could not find container \"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7\": container with ID starting with b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7 not found: ID does not exist"
Sep 16 15:54:14.099071 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.099032    2574 scope.go:117] "RemoveContainer" containerID="b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"
Sep 16 15:54:14.099236 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.099221    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"} err="failed to get container status \"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab\": rpc error: code = NotFound desc = could not find container \"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab\": container with ID starting with b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab not found: ID does not exist"
Sep 16 15:54:14.099272 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.099236    2574 scope.go:117] "RemoveContainer" containerID="2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"
Sep 16 15:54:14.099463 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.099436    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"} err="failed to get container status \"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7\": rpc error: code = NotFound desc = could not find container \"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7\": container with ID starting with 2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7 not found: ID does not exist"
Sep 16 15:54:14.099463 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.099456    2574 scope.go:117] "RemoveContainer" containerID="b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"
Sep 16 15:54:14.099680 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.099665    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"} err="failed to get container status \"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce\": rpc error: code = NotFound desc = could not find container \"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce\": container with ID starting with b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce not found: ID does not exist"
Sep 16 15:54:14.099727 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.099679    2574 scope.go:117] "RemoveContainer" containerID="7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"
Sep 16 15:54:14.099884 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.099869    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"} err="failed to get container status \"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8\": rpc error: code = NotFound desc = could not find container \"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8\": container with ID starting with 7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8 not found: ID does not exist"
Sep 16 15:54:14.099923 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.099883    2574 scope.go:117] "RemoveContainer" containerID="70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"
Sep 16 15:54:14.100045 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.100031    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"} err="failed to get container status \"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7\": rpc error: code = NotFound desc = could not find container \"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7\": container with ID starting with 70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7 not found: ID does not exist"
Sep 16 15:54:14.100094 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.100044    2574 scope.go:117] "RemoveContainer" containerID="799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"
Sep 16 15:54:14.100231 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.100218    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"} err="failed to get container status \"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888\": rpc error: code = NotFound desc = could not find container \"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888\": container with ID starting with 799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888 not found: ID does not exist"
Sep 16 15:54:14.100231 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.100229    2574 scope.go:117] "RemoveContainer" containerID="b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"
Sep 16 15:54:14.100417 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.100400    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"} err="failed to get container status \"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7\": rpc error: code = NotFound desc = could not find container \"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7\": container with ID starting with b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7 not found: ID does not exist"
Sep 16 15:54:14.100459 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.100418    2574 scope.go:117] "RemoveContainer" containerID="b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"
Sep 16 15:54:14.100658 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.100637    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"} err="failed to get container status \"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab\": rpc error: code = NotFound desc = could not find container \"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab\": container with ID starting with b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab not found: ID does not exist"
Sep 16 15:54:14.100704 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.100660    2574 scope.go:117] "RemoveContainer" containerID="2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"
Sep 16 15:54:14.100901 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.100872    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"} err="failed to get container status \"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7\": rpc error: code = NotFound desc = could not find container \"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7\": container with ID starting with 2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7 not found: ID does not exist"
Sep 16 15:54:14.100901 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.100900    2574 scope.go:117] "RemoveContainer" containerID="b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"
Sep 16 15:54:14.101156 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.101130    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"} err="failed to get container status \"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce\": rpc error: code = NotFound desc = could not find container \"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce\": container with ID starting with b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce not found: ID does not exist"
Sep 16 15:54:14.101215 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.101158    2574 scope.go:117] "RemoveContainer" containerID="7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"
Sep 16 15:54:14.101261 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.101236    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.101491 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.101429    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"} err="failed to get container status \"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8\": rpc error: code = NotFound desc = could not find container \"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8\": container with ID starting with 7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8 not found: ID does not exist"
Sep 16 15:54:14.101491 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.101464    2574 scope.go:117] "RemoveContainer" containerID="70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"
Sep 16 15:54:14.101745 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.101721    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"} err="failed to get container status \"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7\": rpc error: code = NotFound desc = could not find container \"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7\": container with ID starting with 70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7 not found: ID does not exist"
Sep 16 15:54:14.101824 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.101746    2574 scope.go:117] "RemoveContainer" containerID="799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"
Sep 16 15:54:14.101995 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.101976    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"} err="failed to get container status \"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888\": rpc error: code = NotFound desc = could not find container \"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888\": container with ID starting with 799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888 not found: ID does not exist"
Sep 16 15:54:14.102060 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.101997    2574 scope.go:117] "RemoveContainer" containerID="b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"
Sep 16 15:54:14.102287 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.102267    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"} err="failed to get container status \"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7\": rpc error: code = NotFound desc = could not find container \"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7\": container with ID starting with b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7 not found: ID does not exist"
Sep 16 15:54:14.102332 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.102288    2574 scope.go:117] "RemoveContainer" containerID="b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"
Sep 16 15:54:14.102545 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.102512    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"} err="failed to get container status \"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab\": rpc error: code = NotFound desc = could not find container \"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab\": container with ID starting with b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab not found: ID does not exist"
Sep 16 15:54:14.102597 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.102548    2574 scope.go:117] "RemoveContainer" containerID="2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"
Sep 16 15:54:14.102834 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.102816    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"} err="failed to get container status \"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7\": rpc error: code = NotFound desc = could not find container \"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7\": container with ID starting with 2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7 not found: ID does not exist"
Sep 16 15:54:14.102887 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.102835    2574 scope.go:117] "RemoveContainer" containerID="b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"
Sep 16 15:54:14.103079 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.103062    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce"} err="failed to get container status \"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce\": rpc error: code = NotFound desc = could not find container \"b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce\": container with ID starting with b760cf994618cbf2fe1aaaf8a95dd51d7e6d95d325f250ced4e716204bd9afce not found: ID does not exist"
Sep 16 15:54:14.103079 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.103079    2574 scope.go:117] "RemoveContainer" containerID="7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"
Sep 16 15:54:14.103323 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.103298    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8"} err="failed to get container status \"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8\": rpc error: code = NotFound desc = could not find container \"7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8\": container with ID starting with 7f4f88e2c04d122dd597d4e965a1903ce9ab2c825a1c4bfba0504551ee3d18c8 not found: ID does not exist"
Sep 16 15:54:14.103323 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.103322    2574 scope.go:117] "RemoveContainer" containerID="70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"
Sep 16 15:54:14.103577 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.103557    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7"} err="failed to get container status \"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7\": rpc error: code = NotFound desc = could not find container \"70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7\": container with ID starting with 70729a726e03da5039253a60331a32682555867cd5427cc71a6f506d496f78f7 not found: ID does not exist"
Sep 16 15:54:14.103577 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.103577    2574 scope.go:117] "RemoveContainer" containerID="799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"
Sep 16 15:54:14.103810 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.103792    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888"} err="failed to get container status \"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888\": rpc error: code = NotFound desc = could not find container \"799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888\": container with ID starting with 799db8a1f63266c1ee22e512a678b12fa74d95118d90ff3b72ed771209262888 not found: ID does not exist"
Sep 16 15:54:14.103859 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.103811    2574 scope.go:117] "RemoveContainer" containerID="b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"
Sep 16 15:54:14.104233 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.103955    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 15:54:14.104233 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.103955    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 15:54:14.104233 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.104042    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 15:54:14.104233 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.103989    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 15:54:14.104233 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.104068    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 15:54:14.104233 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.104043    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 15:54:14.104233 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.103988    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 15:54:14.104233 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.104046    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4pnsn\""
Sep 16 15:54:14.104233 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.104200    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 15:54:14.104663 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.104293    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7"} err="failed to get container status \"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7\": rpc error: code = NotFound desc = could not find container \"b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7\": container with ID starting with b5ed34b40a276fa5eecad8f7ef20d0b65247c592c95363306bbda9e3ecd9fbd7 not found: ID does not exist"
Sep 16 15:54:14.104663 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.104320    2574 scope.go:117] "RemoveContainer" containerID="b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"
Sep 16 15:54:14.104663 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.104450    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 15:54:14.104663 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.104542    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 15:54:14.104663 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.104557    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f6eafte76m100\""
Sep 16 15:54:14.104886 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.104687    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab"} err="failed to get container status \"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab\": rpc error: code = NotFound desc = could not find container \"b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab\": container with ID starting with b338e909cee682cc575775b76d1d7ee6eb537976cdb946ba4484b162b51885ab not found: ID does not exist"
Sep 16 15:54:14.104886 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.104707    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 15:54:14.104886 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.104711    2574 scope.go:117] "RemoveContainer" containerID="2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"
Sep 16 15:54:14.105050 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.105007    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 15:54:14.105050 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.105023    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7"} err="failed to get container status \"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7\": rpc error: code = NotFound desc = could not find container \"2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7\": container with ID starting with 2f244da967aaddc6d9b55c21e4b47fd29435876d0008df838c25185ec419d0b7 not found: ID does not exist"
Sep 16 15:54:14.108386 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.108366    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 15:54:14.111184 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.111153    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 15:54:14.114408 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.114389    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 15:54:14.197213 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.197136    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-web-config\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.197435 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.197416    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-config\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.197556 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.197540    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkt8l\" (UniqueName: \"kubernetes.io/projected/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-kube-api-access-vkt8l\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.197660 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.197646    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.197771 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.197758    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.197870 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.197855    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.197975 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.197962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.198080 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.198066    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.198165 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.198154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.198262 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.198252    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-config-out\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.198348 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.198337    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.198438 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.198427    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.198550 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.198536    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.198655 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.198642    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.198737 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.198727    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.198839 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.198828    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.198925 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.198913    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.199005 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.198993    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.199089 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.199077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.199182 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.199171    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300208 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300170    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-config-out\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300414 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300221    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300414 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300249    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300414 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300269    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300414 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300286    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300414 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300314    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300414 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300354    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300414 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300378    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300414 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300866 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300434    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300866 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300866 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300510    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-web-config\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300866 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300560    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-config\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300866 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkt8l\" (UniqueName: \"kubernetes.io/projected/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-kube-api-access-vkt8l\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300866 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300866 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300648    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300866 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300684    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300866 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300749    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300866 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300806    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.300866 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.300841    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.302107 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.301015    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.302107 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.301253    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.302107 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.301329    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.302107 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.301936    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.302107 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.302090    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.302631 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.302208    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.303090 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.302977    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-config-out\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.303209 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.303188    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.303503 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.303460    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.304118 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.304042    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.304783 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.304761    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.305212 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.305188    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.305417 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.305395    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.305563 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.305543    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.305563 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.305549    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-config\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.305721 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.305681    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.305838 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.305819    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-web-config\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.306479 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.306462    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.306880 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.306862    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.308330 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.308313    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkt8l\" (UniqueName: \"kubernetes.io/projected/c7548b24-c2b9-4da6-a35d-8ba81ee1086c-kube-api-access-vkt8l\") pod \"prometheus-k8s-0\" (UID: \"c7548b24-c2b9-4da6-a35d-8ba81ee1086c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.412013 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.411979    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:54:14.697462 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:14.697437    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 15:54:14.699230 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:54:14.699205    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7548b24_c2b9_4da6_a35d_8ba81ee1086c.slice/crio-f23cf8c54559a172effa0500c7b335ea8b1b8f127f1cd6c1827f240fc20422cc WatchSource:0}: Error finding container f23cf8c54559a172effa0500c7b335ea8b1b8f127f1cd6c1827f240fc20422cc: Status 404 returned error can't find the container with id f23cf8c54559a172effa0500c7b335ea8b1b8f127f1cd6c1827f240fc20422cc
Sep 16 15:54:15.048964 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:15.048891    2574 generic.go:356] "Generic (PLEG): container finished" podID="c7548b24-c2b9-4da6-a35d-8ba81ee1086c" containerID="a947ca6c7af746d8fe8d3843ad9d9a97432fe7cd495138eb45be3e093aaad0ef" exitCode=0
Sep 16 15:54:15.049390 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:15.048981    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7548b24-c2b9-4da6-a35d-8ba81ee1086c","Type":"ContainerDied","Data":"a947ca6c7af746d8fe8d3843ad9d9a97432fe7cd495138eb45be3e093aaad0ef"}
Sep 16 15:54:15.049390 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:15.049058    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7548b24-c2b9-4da6-a35d-8ba81ee1086c","Type":"ContainerStarted","Data":"f23cf8c54559a172effa0500c7b335ea8b1b8f127f1cd6c1827f240fc20422cc"}
Sep 16 15:54:15.817029 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:15.817002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:54:15.819102 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:15.819083    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1332ae3f-698f-4d0e-9801-2aa87121a12b-metrics-certs\") pod \"network-metrics-daemon-bbfmm\" (UID: \"1332ae3f-698f-4d0e-9801-2aa87121a12b\") " pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:54:15.895207 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:15.895139    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mmzn4\""
Sep 16 15:54:15.903785 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:15.903766    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbfmm"
Sep 16 15:54:15.994494 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:15.994460    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="326c729f-9a4e-4cfb-950c-30271d18a0ff" path="/var/lib/kubelet/pods/326c729f-9a4e-4cfb-950c-30271d18a0ff/volumes"
Sep 16 15:54:16.056200 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:16.056170    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn" event={"ID":"86294e16-98ef-4a59-aae0-68600f10abbf","Type":"ContainerStarted","Data":"2ac411776b361f36d9a773b4517337c927ccdd03a271926ee39b405b08bb60cf"}
Sep 16 15:54:16.056669 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:16.056237    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn" event={"ID":"86294e16-98ef-4a59-aae0-68600f10abbf","Type":"ContainerStarted","Data":"36c5eeb6abdaf3d8b903b92b89a0eaec8dd773f1684f9ab73bbbccc465adcedb"}
Sep 16 15:54:16.056669 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:16.056295    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn" event={"ID":"86294e16-98ef-4a59-aae0-68600f10abbf","Type":"ContainerStarted","Data":"324f3e1228ffa82c5dfaeeb309cb67f6e14a61e65d4b4d6557e4172196ba2433"}
Sep 16 15:54:16.059807 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:16.059782    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7548b24-c2b9-4da6-a35d-8ba81ee1086c","Type":"ContainerStarted","Data":"7432b43187428aeae5b69390049c14c23b595bb5bef75a970c837b5b068545cb"}
Sep 16 15:54:16.059908 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:16.059850    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7548b24-c2b9-4da6-a35d-8ba81ee1086c","Type":"ContainerStarted","Data":"9d64c7ee874877eb7f059b3ffcd3193346a6334c16a14eb7bdf6ca92359eef9c"}
Sep 16 15:54:16.059908 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:16.059888    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7548b24-c2b9-4da6-a35d-8ba81ee1086c","Type":"ContainerStarted","Data":"7d7512a0ce96639ae1f276f4496973bd3fbe12b881a81f839e2a8ea4625abcca"}
Sep 16 15:54:16.060000 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:16.059924    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7548b24-c2b9-4da6-a35d-8ba81ee1086c","Type":"ContainerStarted","Data":"878634de1de5bebbfecb8fa3b0ef7686ffebeaf0117f7bea4db956a2af7819d8"}
Sep 16 15:54:16.060000 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:16.059960    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7548b24-c2b9-4da6-a35d-8ba81ee1086c","Type":"ContainerStarted","Data":"ca827011c0d52ed86cfd7112ed471b57fe1d0577841e4fb00257a2a35d5103a3"}
Sep 16 15:54:16.060074 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:16.059999    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c7548b24-c2b9-4da6-a35d-8ba81ee1086c","Type":"ContainerStarted","Data":"99723cc2979a2ed921a0d6b6e071a1b61e1aa8a1823adfb4fe3972ab708e12f8"}
Sep 16 15:54:16.078967 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:16.077514    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6786f7c4c4-8srfn" podStartSLOduration=1.4601185970000001 podStartE2EDuration="3.077494651s" totalImagesPullingTime="1.617376054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:54:16.075466219 +0000 UTC m=+252.595088998" watchObservedRunningTime="2026-09-16 15:54:16.077494651 +0000 UTC m=+252.597117428"
Sep 16 15:54:16.104978 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:16.104917    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.083236946 podStartE2EDuration="2.104898644s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.661698ms" isStatefulPod=false podCreationTimestamp="2026-09-16 15:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:54:16.103335805 +0000 UTC m=+252.622958613" watchObservedRunningTime="2026-09-16 15:54:16.104898644 +0000 UTC m=+252.624521422"
Sep 16 15:54:16.169795 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:16.169757    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bbfmm"]
Sep 16 15:54:16.172247 ip-10-0-143-143 kubenswrapper[2574]: W0916 15:54:16.172222    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1332ae3f_698f_4d0e_9801_2aa87121a12b.slice/crio-ad34a43cb02c10365461068f1c280110c62d3fd21abe1b44b2958292a2e48fd4 WatchSource:0}: Error finding container ad34a43cb02c10365461068f1c280110c62d3fd21abe1b44b2958292a2e48fd4: Status 404 returned error can't find the container with id ad34a43cb02c10365461068f1c280110c62d3fd21abe1b44b2958292a2e48fd4
Sep 16 15:54:17.066703 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:17.066610    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bbfmm" event={"ID":"1332ae3f-698f-4d0e-9801-2aa87121a12b","Type":"ContainerStarted","Data":"ad34a43cb02c10365461068f1c280110c62d3fd21abe1b44b2958292a2e48fd4"}
Sep 16 15:54:18.072710 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:18.072672    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bbfmm" event={"ID":"1332ae3f-698f-4d0e-9801-2aa87121a12b","Type":"ContainerStarted","Data":"f514342a07c51459751e09b2f43e4c142805c854417d8b914341a70b3dbcad30"}
Sep 16 15:54:18.073157 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:18.072737    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bbfmm" event={"ID":"1332ae3f-698f-4d0e-9801-2aa87121a12b","Type":"ContainerStarted","Data":"48a629fdf4f2bebb879710466c2e47ce5ff42fa124a0935c49685b508957fcc2"}
Sep 16 15:54:18.087762 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:18.087709    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bbfmm" podStartSLOduration=253.229490659 podStartE2EDuration="4m14.087693012s" totalImagesPullingTime="858.202353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:54:18.087189654 +0000 UTC m=+254.606812431" watchObservedRunningTime="2026-09-16 15:54:18.087693012 +0000 UTC m=+254.607315788"
Sep 16 15:54:19.413145 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:54:19.413089    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:55:03.920546 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:55:03.920504    2574 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 15:55:14.412739 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:55:14.412682    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:55:14.428700 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:55:14.428667    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 15:55:15.344110 ip-10-0-143-143 kubenswrapper[2574]: I0916 15:55:15.344077    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 16:00:19.426895 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:19.426863    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-x7j9h"]
Sep 16 16:00:19.430367 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:19.430351    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-x7j9h"
Sep 16 16:00:19.432449 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:19.432423    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 16:00:19.432556 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:19.432505    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 16:00:19.433311 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:19.433288    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ntcx4\""
Sep 16 16:00:19.442198 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:19.442168    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-x7j9h"]
Sep 16 16:00:19.537921 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:19.537884    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/660f205d-bd5c-4133-9502-4422b6a210c6-bound-sa-token\") pod \"cert-manager-55b4d954d5-x7j9h\" (UID: \"660f205d-bd5c-4133-9502-4422b6a210c6\") " pod="cert-manager/cert-manager-55b4d954d5-x7j9h"
Sep 16 16:00:19.537921 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:19.537925    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h986\" (UniqueName: \"kubernetes.io/projected/660f205d-bd5c-4133-9502-4422b6a210c6-kube-api-access-7h986\") pod \"cert-manager-55b4d954d5-x7j9h\" (UID: \"660f205d-bd5c-4133-9502-4422b6a210c6\") " pod="cert-manager/cert-manager-55b4d954d5-x7j9h"
Sep 16 16:00:19.638445 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:19.638408    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/660f205d-bd5c-4133-9502-4422b6a210c6-bound-sa-token\") pod \"cert-manager-55b4d954d5-x7j9h\" (UID: \"660f205d-bd5c-4133-9502-4422b6a210c6\") " pod="cert-manager/cert-manager-55b4d954d5-x7j9h"
Sep 16 16:00:19.638664 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:19.638458    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h986\" (UniqueName: \"kubernetes.io/projected/660f205d-bd5c-4133-9502-4422b6a210c6-kube-api-access-7h986\") pod \"cert-manager-55b4d954d5-x7j9h\" (UID: \"660f205d-bd5c-4133-9502-4422b6a210c6\") " pod="cert-manager/cert-manager-55b4d954d5-x7j9h"
Sep 16 16:00:19.646498 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:19.646458    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/660f205d-bd5c-4133-9502-4422b6a210c6-bound-sa-token\") pod \"cert-manager-55b4d954d5-x7j9h\" (UID: \"660f205d-bd5c-4133-9502-4422b6a210c6\") " pod="cert-manager/cert-manager-55b4d954d5-x7j9h"
Sep 16 16:00:19.646617 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:19.646464    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h986\" (UniqueName: \"kubernetes.io/projected/660f205d-bd5c-4133-9502-4422b6a210c6-kube-api-access-7h986\") pod \"cert-manager-55b4d954d5-x7j9h\" (UID: \"660f205d-bd5c-4133-9502-4422b6a210c6\") " pod="cert-manager/cert-manager-55b4d954d5-x7j9h"
Sep 16 16:00:19.754413 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:19.754323    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-x7j9h"
Sep 16 16:00:20.022501 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:20.022473    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-x7j9h"]
Sep 16 16:00:20.024955 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:00:20.024928    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod660f205d_bd5c_4133_9502_4422b6a210c6.slice/crio-3f2dfa1e89071c133b32e64dd03978e50d9e5c3e87be40e02404c7672c9edc01 WatchSource:0}: Error finding container 3f2dfa1e89071c133b32e64dd03978e50d9e5c3e87be40e02404c7672c9edc01: Status 404 returned error can't find the container with id 3f2dfa1e89071c133b32e64dd03978e50d9e5c3e87be40e02404c7672c9edc01
Sep 16 16:00:20.027016 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:20.026996    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 16:00:20.573303 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:20.573267    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-x7j9h" event={"ID":"660f205d-bd5c-4133-9502-4422b6a210c6","Type":"ContainerStarted","Data":"3f2dfa1e89071c133b32e64dd03978e50d9e5c3e87be40e02404c7672c9edc01"}
Sep 16 16:00:23.589125 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:23.589036    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-x7j9h" event={"ID":"660f205d-bd5c-4133-9502-4422b6a210c6","Type":"ContainerStarted","Data":"504fd07e9806fdb2cc307e99ef01efc1ca703f400aecadabbab474807ceaed72"}
Sep 16 16:00:23.607375 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:23.607322    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-x7j9h" podStartSLOduration=1.498887982 podStartE2EDuration="4.607304157s" totalImagesPullingTime="3.108416175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:00:23.605862093 +0000 UTC m=+620.125484891" watchObservedRunningTime="2026-09-16 16:00:23.607304157 +0000 UTC m=+620.126926934"
Sep 16 16:00:46.421187 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.421147    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"]
Sep 16 16:00:46.427641 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.427618    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:46.430071 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.430030    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 16:00:46.430883 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.430862    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 16:00:46.431026 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.430868    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 16:00:46.431026 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.430868    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 16:00:46.431217 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.431180    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 16:00:46.431289 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.431246    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rxqwh\""
Sep 16 16:00:46.438075 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.438053    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"]
Sep 16 16:00:46.474127 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.474085    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2b5ce5e-09a1-450f-8636-86e76195e6f5-cert\") pod \"lws-controller-manager-75c8bf98c8-dvlgl\" (UID: \"a2b5ce5e-09a1-450f-8636-86e76195e6f5\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:46.474306 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.474147    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2b5ce5e-09a1-450f-8636-86e76195e6f5-metrics-cert\") pod \"lws-controller-manager-75c8bf98c8-dvlgl\" (UID: \"a2b5ce5e-09a1-450f-8636-86e76195e6f5\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:46.474306 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.474199    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2b5ce5e-09a1-450f-8636-86e76195e6f5-manager-config\") pod \"lws-controller-manager-75c8bf98c8-dvlgl\" (UID: \"a2b5ce5e-09a1-450f-8636-86e76195e6f5\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:46.474306 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.474222    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnpbj\" (UniqueName: \"kubernetes.io/projected/a2b5ce5e-09a1-450f-8636-86e76195e6f5-kube-api-access-dnpbj\") pod \"lws-controller-manager-75c8bf98c8-dvlgl\" (UID: \"a2b5ce5e-09a1-450f-8636-86e76195e6f5\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:46.574989 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.574948    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2b5ce5e-09a1-450f-8636-86e76195e6f5-manager-config\") pod \"lws-controller-manager-75c8bf98c8-dvlgl\" (UID: \"a2b5ce5e-09a1-450f-8636-86e76195e6f5\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:46.575178 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.574999    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnpbj\" (UniqueName: \"kubernetes.io/projected/a2b5ce5e-09a1-450f-8636-86e76195e6f5-kube-api-access-dnpbj\") pod \"lws-controller-manager-75c8bf98c8-dvlgl\" (UID: \"a2b5ce5e-09a1-450f-8636-86e76195e6f5\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:46.575178 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.575040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2b5ce5e-09a1-450f-8636-86e76195e6f5-cert\") pod \"lws-controller-manager-75c8bf98c8-dvlgl\" (UID: \"a2b5ce5e-09a1-450f-8636-86e76195e6f5\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:46.575178 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.575085    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2b5ce5e-09a1-450f-8636-86e76195e6f5-metrics-cert\") pod \"lws-controller-manager-75c8bf98c8-dvlgl\" (UID: \"a2b5ce5e-09a1-450f-8636-86e76195e6f5\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:46.575801 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.575781    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2b5ce5e-09a1-450f-8636-86e76195e6f5-manager-config\") pod \"lws-controller-manager-75c8bf98c8-dvlgl\" (UID: \"a2b5ce5e-09a1-450f-8636-86e76195e6f5\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:46.577553 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.577533    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2b5ce5e-09a1-450f-8636-86e76195e6f5-cert\") pod \"lws-controller-manager-75c8bf98c8-dvlgl\" (UID: \"a2b5ce5e-09a1-450f-8636-86e76195e6f5\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:46.577661 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.577641    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2b5ce5e-09a1-450f-8636-86e76195e6f5-metrics-cert\") pod \"lws-controller-manager-75c8bf98c8-dvlgl\" (UID: \"a2b5ce5e-09a1-450f-8636-86e76195e6f5\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:46.587425 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.587392    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnpbj\" (UniqueName: \"kubernetes.io/projected/a2b5ce5e-09a1-450f-8636-86e76195e6f5-kube-api-access-dnpbj\") pod \"lws-controller-manager-75c8bf98c8-dvlgl\" (UID: \"a2b5ce5e-09a1-450f-8636-86e76195e6f5\") " pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:46.737254 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:46.737164    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:47.007893 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:47.007869    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"]
Sep 16 16:00:47.010792 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:00:47.010761    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b5ce5e_09a1_450f_8636_86e76195e6f5.slice/crio-c1ad643b6033666550c1ea829d1e752e8939f5468169b8c23c54bb940fa95656 WatchSource:0}: Error finding container c1ad643b6033666550c1ea829d1e752e8939f5468169b8c23c54bb940fa95656: Status 404 returned error can't find the container with id c1ad643b6033666550c1ea829d1e752e8939f5468169b8c23c54bb940fa95656
Sep 16 16:00:47.693372 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:47.693332    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl" event={"ID":"a2b5ce5e-09a1-450f-8636-86e76195e6f5","Type":"ContainerStarted","Data":"c1ad643b6033666550c1ea829d1e752e8939f5468169b8c23c54bb940fa95656"}
Sep 16 16:00:50.421927 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.421890    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"]
Sep 16 16:00:50.427594 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.427570    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.431919 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.431893    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 16:00:50.432023 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.431943    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-bvvsr\""
Sep 16 16:00:50.439584 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.439558    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"]
Sep 16 16:00:50.509958 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.509923    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.509958 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.509957    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.510169 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.509978    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9980823-1d5e-4996-8eec-7054d9091b82-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.510169 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.509999    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwzbn\" (UniqueName: \"kubernetes.io/projected/f9980823-1d5e-4996-8eec-7054d9091b82-kube-api-access-gwzbn\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.510169 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.510081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.510169 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.510156    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9980823-1d5e-4996-8eec-7054d9091b82-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.510317 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.510183    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9980823-1d5e-4996-8eec-7054d9091b82-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.510317 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.510207    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.510317 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.510224    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.610913 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.610879    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9980823-1d5e-4996-8eec-7054d9091b82-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.611093 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.610936    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9980823-1d5e-4996-8eec-7054d9091b82-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.611093 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.610970    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.611093 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.610989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.611093 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.611046    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.611093 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.611073    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.611367 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.611100    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9980823-1d5e-4996-8eec-7054d9091b82-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.611367 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.611134    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzbn\" (UniqueName: \"kubernetes.io/projected/f9980823-1d5e-4996-8eec-7054d9091b82-kube-api-access-gwzbn\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.611367 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.611289    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.611641 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.611471    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.611785 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.611763    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.611849 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.611818    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.611849 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.611838    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.611940 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.611860    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9980823-1d5e-4996-8eec-7054d9091b82-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.613295 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.613279    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9980823-1d5e-4996-8eec-7054d9091b82-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.613539 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.613504    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9980823-1d5e-4996-8eec-7054d9091b82-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.619199 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.619170    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9980823-1d5e-4996-8eec-7054d9091b82-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.619318 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.619301    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwzbn\" (UniqueName: \"kubernetes.io/projected/f9980823-1d5e-4996-8eec-7054d9091b82-kube-api-access-gwzbn\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9582c\" (UID: \"f9980823-1d5e-4996-8eec-7054d9091b82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:50.740724 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:50.740644    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:51.018265 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:51.018242    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"]
Sep 16 16:00:51.020573 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:00:51.020546    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9980823_1d5e_4996_8eec_7054d9091b82.slice/crio-1b7fd397477d37b252b9baef2171db38549bbbcce729294bcef2183dcbeb1a7c WatchSource:0}: Error finding container 1b7fd397477d37b252b9baef2171db38549bbbcce729294bcef2183dcbeb1a7c: Status 404 returned error can't find the container with id 1b7fd397477d37b252b9baef2171db38549bbbcce729294bcef2183dcbeb1a7c
Sep 16 16:00:51.717231 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:51.717196    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c" event={"ID":"f9980823-1d5e-4996-8eec-7054d9091b82","Type":"ContainerStarted","Data":"1b7fd397477d37b252b9baef2171db38549bbbcce729294bcef2183dcbeb1a7c"}
Sep 16 16:00:55.183090 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:55.183049    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 16:00:55.183440 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:55.183128    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 16:00:55.183440 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:55.183161    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 16:00:55.739031 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:55.738983    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl" event={"ID":"a2b5ce5e-09a1-450f-8636-86e76195e6f5","Type":"ContainerStarted","Data":"4ba26b1eb9ae945f0b0b80e126d9954370110bdc8490784228d79c3ed042c889"}
Sep 16 16:00:55.739233 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:55.739061    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:00:55.740597 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:55.740571    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c" event={"ID":"f9980823-1d5e-4996-8eec-7054d9091b82","Type":"ContainerStarted","Data":"a219d90dc0887343cec8a339120a5847ddb094f16ac55b6c17f1457d70bbdeef"}
Sep 16 16:00:55.740717 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:55.740697    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:55.756836 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:55.756792    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl" podStartSLOduration=1.637833576 podStartE2EDuration="9.756778718s" totalImagesPullingTime="8.118945142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:00:55.755148736 +0000 UTC m=+652.274771507" watchObservedRunningTime="2026-09-16 16:00:55.756778718 +0000 UTC m=+652.276401495"
Sep 16 16:00:55.774049 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:55.774008    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c" podStartSLOduration=1.613958032 podStartE2EDuration="5.773995179s" totalImagesPullingTime="4.160037147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:00:55.773655491 +0000 UTC m=+652.293278275" watchObservedRunningTime="2026-09-16 16:00:55.773995179 +0000 UTC m=+652.293617955"
Sep 16 16:00:56.745994 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:56.745967    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:57.749298 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:57.749270    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:00:57.750357 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:00:57.750338    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:01:06.747857 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:06.747826    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-75c8bf98c8-dvlgl"
Sep 16 16:01:34.650719 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:34.650684    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wdgxz"]
Sep 16 16:01:34.654071 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:34.654053    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wdgxz"
Sep 16 16:01:34.656959 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:34.656937    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 16:01:34.657486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:34.657470    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 16:01:34.658741 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:34.658724    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7knk6\""
Sep 16 16:01:34.679303 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:34.679278    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wdgxz"]
Sep 16 16:01:34.697117 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:34.697081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdnsp\" (UniqueName: \"kubernetes.io/projected/5288fd6d-8a27-4941-917c-f8cf29e38b2e-kube-api-access-hdnsp\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-wdgxz\" (UID: \"5288fd6d-8a27-4941-917c-f8cf29e38b2e\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wdgxz"
Sep 16 16:01:34.798464 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:34.798422    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnsp\" (UniqueName: \"kubernetes.io/projected/5288fd6d-8a27-4941-917c-f8cf29e38b2e-kube-api-access-hdnsp\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-wdgxz\" (UID: \"5288fd6d-8a27-4941-917c-f8cf29e38b2e\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wdgxz"
Sep 16 16:01:34.810356 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:34.810332    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdnsp\" (UniqueName: \"kubernetes.io/projected/5288fd6d-8a27-4941-917c-f8cf29e38b2e-kube-api-access-hdnsp\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-wdgxz\" (UID: \"5288fd6d-8a27-4941-917c-f8cf29e38b2e\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wdgxz"
Sep 16 16:01:34.963456 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:34.963362    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wdgxz"
Sep 16 16:01:35.233809 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:35.233737    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wdgxz"]
Sep 16 16:01:35.237108 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:01:35.237083    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5288fd6d_8a27_4941_917c_f8cf29e38b2e.slice/crio-ffa4aed3beaf5205986253bab671345d95a46e5d370393a30b46927edb0a1b5d WatchSource:0}: Error finding container ffa4aed3beaf5205986253bab671345d95a46e5d370393a30b46927edb0a1b5d: Status 404 returned error can't find the container with id ffa4aed3beaf5205986253bab671345d95a46e5d370393a30b46927edb0a1b5d
Sep 16 16:01:35.916441 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:35.916405    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wdgxz" event={"ID":"5288fd6d-8a27-4941-917c-f8cf29e38b2e","Type":"ContainerStarted","Data":"ffa4aed3beaf5205986253bab671345d95a46e5d370393a30b46927edb0a1b5d"}
Sep 16 16:01:38.933737 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:38.933702    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wdgxz" event={"ID":"5288fd6d-8a27-4941-917c-f8cf29e38b2e","Type":"ContainerStarted","Data":"0a49db5a273322e781e1f5c0a7795947162e9d422d8b9a4ad01e3f4da18bd630"}
Sep 16 16:01:38.934133 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:38.933819    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wdgxz"
Sep 16 16:01:38.957842 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:38.957784    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wdgxz" podStartSLOduration=1.905519634 podStartE2EDuration="4.957766619s" totalImagesPullingTime="3.052246985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:01:38.956230232 +0000 UTC m=+695.475853009" watchObservedRunningTime="2026-09-16 16:01:38.957766619 +0000 UTC m=+695.477389395"
Sep 16 16:01:49.941341 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:01:49.941307    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wdgxz"
Sep 16 16:02:21.561327 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:21.561291    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-l6kvv"]
Sep 16 16:02:21.569341 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:21.569315    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-l6kvv"
Sep 16 16:02:21.571346 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:21.571316    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-l6kvv"]
Sep 16 16:02:21.571891 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:21.571870    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wrjb7\""
Sep 16 16:02:21.636332 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:21.636285    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qznf\" (UniqueName: \"kubernetes.io/projected/a5aa3b57-d9b8-49ab-b017-484884682f7d-kube-api-access-5qznf\") pod \"authorino-7bcfd8f9cd-l6kvv\" (UID: \"a5aa3b57-d9b8-49ab-b017-484884682f7d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-l6kvv"
Sep 16 16:02:21.730107 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:21.730068    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-4mgz2"]
Sep 16 16:02:21.733965 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:21.733942    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4mgz2"
Sep 16 16:02:21.737049 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:21.737023    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qznf\" (UniqueName: \"kubernetes.io/projected/a5aa3b57-d9b8-49ab-b017-484884682f7d-kube-api-access-5qznf\") pod \"authorino-7bcfd8f9cd-l6kvv\" (UID: \"a5aa3b57-d9b8-49ab-b017-484884682f7d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-l6kvv"
Sep 16 16:02:21.740427 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:21.740402    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4mgz2"]
Sep 16 16:02:21.746628 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:21.746599    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qznf\" (UniqueName: \"kubernetes.io/projected/a5aa3b57-d9b8-49ab-b017-484884682f7d-kube-api-access-5qznf\") pod \"authorino-7bcfd8f9cd-l6kvv\" (UID: \"a5aa3b57-d9b8-49ab-b017-484884682f7d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-l6kvv"
Sep 16 16:02:21.838037 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:21.837942    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn94t\" (UniqueName: \"kubernetes.io/projected/2fca3b85-4761-4c84-88f2-29f548f121e7-kube-api-access-zn94t\") pod \"authorino-59c974bd48-4mgz2\" (UID: \"2fca3b85-4761-4c84-88f2-29f548f121e7\") " pod="kuadrant-system/authorino-59c974bd48-4mgz2"
Sep 16 16:02:21.878956 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:21.878920    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-l6kvv"
Sep 16 16:02:21.939257 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:21.939188    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn94t\" (UniqueName: \"kubernetes.io/projected/2fca3b85-4761-4c84-88f2-29f548f121e7-kube-api-access-zn94t\") pod \"authorino-59c974bd48-4mgz2\" (UID: \"2fca3b85-4761-4c84-88f2-29f548f121e7\") " pod="kuadrant-system/authorino-59c974bd48-4mgz2"
Sep 16 16:02:21.948043 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:21.948017    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn94t\" (UniqueName: \"kubernetes.io/projected/2fca3b85-4761-4c84-88f2-29f548f121e7-kube-api-access-zn94t\") pod \"authorino-59c974bd48-4mgz2\" (UID: \"2fca3b85-4761-4c84-88f2-29f548f121e7\") " pod="kuadrant-system/authorino-59c974bd48-4mgz2"
Sep 16 16:02:22.043949 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:22.043915    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4mgz2"
Sep 16 16:02:22.151028 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:22.151000    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-l6kvv"]
Sep 16 16:02:22.153710 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:02:22.153683    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5aa3b57_d9b8_49ab_b017_484884682f7d.slice/crio-5c03533c42c924ee3bed2840d28a336684488dfad7e06aba1a6a6acbc7307c9f WatchSource:0}: Error finding container 5c03533c42c924ee3bed2840d28a336684488dfad7e06aba1a6a6acbc7307c9f: Status 404 returned error can't find the container with id 5c03533c42c924ee3bed2840d28a336684488dfad7e06aba1a6a6acbc7307c9f
Sep 16 16:02:22.314817 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:22.314785    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4mgz2"]
Sep 16 16:02:22.317818 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:02:22.317791    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fca3b85_4761_4c84_88f2_29f548f121e7.slice/crio-49b07e2f4d32e14c84aba923a573ff6b8ecf29f7f8a1be473d4bb8be7c1f6c78 WatchSource:0}: Error finding container 49b07e2f4d32e14c84aba923a573ff6b8ecf29f7f8a1be473d4bb8be7c1f6c78: Status 404 returned error can't find the container with id 49b07e2f4d32e14c84aba923a573ff6b8ecf29f7f8a1be473d4bb8be7c1f6c78
Sep 16 16:02:23.129658 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:23.129624    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4mgz2" event={"ID":"2fca3b85-4761-4c84-88f2-29f548f121e7","Type":"ContainerStarted","Data":"49b07e2f4d32e14c84aba923a573ff6b8ecf29f7f8a1be473d4bb8be7c1f6c78"}
Sep 16 16:02:23.130881 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:23.130853    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-l6kvv" event={"ID":"a5aa3b57-d9b8-49ab-b017-484884682f7d","Type":"ContainerStarted","Data":"5c03533c42c924ee3bed2840d28a336684488dfad7e06aba1a6a6acbc7307c9f"}
Sep 16 16:02:25.144120 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:25.144088    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4mgz2" event={"ID":"2fca3b85-4761-4c84-88f2-29f548f121e7","Type":"ContainerStarted","Data":"771af7306acca6f28352bceef17aab07126fd14e091d096881ab5601ae0b03cc"}
Sep 16 16:02:25.161816 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:25.161753    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-4mgz2" podStartSLOduration=1.5023548770000001 podStartE2EDuration="4.161739088s" totalImagesPullingTime="2.659384211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:02:25.161689646 +0000 UTC m=+741.681312425" watchObservedRunningTime="2026-09-16 16:02:25.161739088 +0000 UTC m=+741.681361865"
Sep 16 16:02:25.192725 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:25.192688    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-l6kvv"]
Sep 16 16:02:26.149852 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:26.149813    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-l6kvv" event={"ID":"a5aa3b57-d9b8-49ab-b017-484884682f7d","Type":"ContainerStarted","Data":"f3f64f38abb1e4e378333b0f253af255e9c9c7ad8914e8d47f93f8038a29e993"}
Sep 16 16:02:26.150367 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:26.149920    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-l6kvv" podUID="a5aa3b57-d9b8-49ab-b017-484884682f7d" containerName="authorino" containerID="cri-o://f3f64f38abb1e4e378333b0f253af255e9c9c7ad8914e8d47f93f8038a29e993" gracePeriod=30
Sep 16 16:02:26.164691 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:26.164651    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-l6kvv" podStartSLOduration=1.5085897350000002 podStartE2EDuration="5.164635884s" totalImagesPullingTime="3.656046149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:02:26.163414871 +0000 UTC m=+742.683037650" watchObservedRunningTime="2026-09-16 16:02:26.164635884 +0000 UTC m=+742.684258661"
Sep 16 16:02:26.399112 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:26.399083    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-l6kvv"
Sep 16 16:02:26.480382 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:26.480286    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qznf\" (UniqueName: \"kubernetes.io/projected/a5aa3b57-d9b8-49ab-b017-484884682f7d-kube-api-access-5qznf\") pod \"a5aa3b57-d9b8-49ab-b017-484884682f7d\" (UID: \"a5aa3b57-d9b8-49ab-b017-484884682f7d\") "
Sep 16 16:02:26.482500 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:26.482475    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5aa3b57-d9b8-49ab-b017-484884682f7d-kube-api-access-5qznf" (OuterVolumeSpecName: "kube-api-access-5qznf") pod "a5aa3b57-d9b8-49ab-b017-484884682f7d" (UID: "a5aa3b57-d9b8-49ab-b017-484884682f7d"). InnerVolumeSpecName "kube-api-access-5qznf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:02:26.581887 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:26.581846    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qznf\" (UniqueName: \"kubernetes.io/projected/a5aa3b57-d9b8-49ab-b017-484884682f7d-kube-api-access-5qznf\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:02:27.156141 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:27.156109    2574 generic.go:356] "Generic (PLEG): container finished" podID="a5aa3b57-d9b8-49ab-b017-484884682f7d" containerID="f3f64f38abb1e4e378333b0f253af255e9c9c7ad8914e8d47f93f8038a29e993" exitCode=0
Sep 16 16:02:27.156546 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:27.156151    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-l6kvv"
Sep 16 16:02:27.156546 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:27.156163    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-l6kvv" event={"ID":"a5aa3b57-d9b8-49ab-b017-484884682f7d","Type":"ContainerDied","Data":"f3f64f38abb1e4e378333b0f253af255e9c9c7ad8914e8d47f93f8038a29e993"}
Sep 16 16:02:27.156546 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:27.156196    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-l6kvv" event={"ID":"a5aa3b57-d9b8-49ab-b017-484884682f7d","Type":"ContainerDied","Data":"5c03533c42c924ee3bed2840d28a336684488dfad7e06aba1a6a6acbc7307c9f"}
Sep 16 16:02:27.156546 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:27.156213    2574 scope.go:117] "RemoveContainer" containerID="f3f64f38abb1e4e378333b0f253af255e9c9c7ad8914e8d47f93f8038a29e993"
Sep 16 16:02:27.164838 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:27.164821    2574 scope.go:117] "RemoveContainer" containerID="f3f64f38abb1e4e378333b0f253af255e9c9c7ad8914e8d47f93f8038a29e993"
Sep 16 16:02:27.165117 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:02:27.165098    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f64f38abb1e4e378333b0f253af255e9c9c7ad8914e8d47f93f8038a29e993\": container with ID starting with f3f64f38abb1e4e378333b0f253af255e9c9c7ad8914e8d47f93f8038a29e993 not found: ID does not exist" containerID="f3f64f38abb1e4e378333b0f253af255e9c9c7ad8914e8d47f93f8038a29e993"
Sep 16 16:02:27.165155 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:27.165127    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f64f38abb1e4e378333b0f253af255e9c9c7ad8914e8d47f93f8038a29e993"} err="failed to get container status \"f3f64f38abb1e4e378333b0f253af255e9c9c7ad8914e8d47f93f8038a29e993\": rpc error: code = NotFound desc = could not find container \"f3f64f38abb1e4e378333b0f253af255e9c9c7ad8914e8d47f93f8038a29e993\": container with ID starting with f3f64f38abb1e4e378333b0f253af255e9c9c7ad8914e8d47f93f8038a29e993 not found: ID does not exist"
Sep 16 16:02:27.175696 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:27.175675    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-l6kvv"]
Sep 16 16:02:27.180976 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:27.180956    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-l6kvv"]
Sep 16 16:02:27.994452 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:27.994420    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5aa3b57-d9b8-49ab-b017-484884682f7d" path="/var/lib/kubelet/pods/a5aa3b57-d9b8-49ab-b017-484884682f7d/volumes"
Sep 16 16:02:46.621563 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:46.621499    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4mgz2"]
Sep 16 16:02:46.622029 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:46.621713    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-4mgz2" podUID="2fca3b85-4761-4c84-88f2-29f548f121e7" containerName="authorino" containerID="cri-o://771af7306acca6f28352bceef17aab07126fd14e091d096881ab5601ae0b03cc" gracePeriod=30
Sep 16 16:02:46.872744 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:46.872680    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4mgz2"
Sep 16 16:02:46.964245 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:46.964213    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn94t\" (UniqueName: \"kubernetes.io/projected/2fca3b85-4761-4c84-88f2-29f548f121e7-kube-api-access-zn94t\") pod \"2fca3b85-4761-4c84-88f2-29f548f121e7\" (UID: \"2fca3b85-4761-4c84-88f2-29f548f121e7\") "
Sep 16 16:02:46.966254 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:46.966227    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fca3b85-4761-4c84-88f2-29f548f121e7-kube-api-access-zn94t" (OuterVolumeSpecName: "kube-api-access-zn94t") pod "2fca3b85-4761-4c84-88f2-29f548f121e7" (UID: "2fca3b85-4761-4c84-88f2-29f548f121e7"). InnerVolumeSpecName "kube-api-access-zn94t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:02:47.065164 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:47.065122    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn94t\" (UniqueName: \"kubernetes.io/projected/2fca3b85-4761-4c84-88f2-29f548f121e7-kube-api-access-zn94t\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:02:47.252700 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:47.252610    2574 generic.go:356] "Generic (PLEG): container finished" podID="2fca3b85-4761-4c84-88f2-29f548f121e7" containerID="771af7306acca6f28352bceef17aab07126fd14e091d096881ab5601ae0b03cc" exitCode=0
Sep 16 16:02:47.252700 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:47.252663    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4mgz2"
Sep 16 16:02:47.252872 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:47.252703    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4mgz2" event={"ID":"2fca3b85-4761-4c84-88f2-29f548f121e7","Type":"ContainerDied","Data":"771af7306acca6f28352bceef17aab07126fd14e091d096881ab5601ae0b03cc"}
Sep 16 16:02:47.252872 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:47.252752    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4mgz2" event={"ID":"2fca3b85-4761-4c84-88f2-29f548f121e7","Type":"ContainerDied","Data":"49b07e2f4d32e14c84aba923a573ff6b8ecf29f7f8a1be473d4bb8be7c1f6c78"}
Sep 16 16:02:47.252872 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:47.252773    2574 scope.go:117] "RemoveContainer" containerID="771af7306acca6f28352bceef17aab07126fd14e091d096881ab5601ae0b03cc"
Sep 16 16:02:47.261589 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:47.261565    2574 scope.go:117] "RemoveContainer" containerID="771af7306acca6f28352bceef17aab07126fd14e091d096881ab5601ae0b03cc"
Sep 16 16:02:47.261870 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:02:47.261850    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771af7306acca6f28352bceef17aab07126fd14e091d096881ab5601ae0b03cc\": container with ID starting with 771af7306acca6f28352bceef17aab07126fd14e091d096881ab5601ae0b03cc not found: ID does not exist" containerID="771af7306acca6f28352bceef17aab07126fd14e091d096881ab5601ae0b03cc"
Sep 16 16:02:47.261933 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:47.261883    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771af7306acca6f28352bceef17aab07126fd14e091d096881ab5601ae0b03cc"} err="failed to get container status \"771af7306acca6f28352bceef17aab07126fd14e091d096881ab5601ae0b03cc\": rpc error: code = NotFound desc = could not find container \"771af7306acca6f28352bceef17aab07126fd14e091d096881ab5601ae0b03cc\": container with ID starting with 771af7306acca6f28352bceef17aab07126fd14e091d096881ab5601ae0b03cc not found: ID does not exist"
Sep 16 16:02:47.273816 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:47.273783    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4mgz2"]
Sep 16 16:02:47.279141 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:47.279108    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4mgz2"]
Sep 16 16:02:47.994682 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:02:47.994652    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fca3b85-4761-4c84-88f2-29f548f121e7" path="/var/lib/kubelet/pods/2fca3b85-4761-4c84-88f2-29f548f121e7/volumes"
Sep 16 16:03:14.089104 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.089069    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6c85c996cc-nw4bw"]
Sep 16 16:03:14.089539 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.089464    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5aa3b57-d9b8-49ab-b017-484884682f7d" containerName="authorino"
Sep 16 16:03:14.089539 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.089476    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5aa3b57-d9b8-49ab-b017-484884682f7d" containerName="authorino"
Sep 16 16:03:14.089539 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.089490    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fca3b85-4761-4c84-88f2-29f548f121e7" containerName="authorino"
Sep 16 16:03:14.089539 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.089496    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fca3b85-4761-4c84-88f2-29f548f121e7" containerName="authorino"
Sep 16 16:03:14.089690 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.089577    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5aa3b57-d9b8-49ab-b017-484884682f7d" containerName="authorino"
Sep 16 16:03:14.089690 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.089586    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fca3b85-4761-4c84-88f2-29f548f121e7" containerName="authorino"
Sep 16 16:03:14.093816 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.093800    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6c85c996cc-nw4bw"
Sep 16 16:03:14.096043 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.096019    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 16 16:03:14.096229 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.096205    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-4d27q\""
Sep 16 16:03:14.096802 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.096780    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 16:03:14.096802 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.096793    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 16:03:14.101206 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.101186    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6c85c996cc-nw4bw"]
Sep 16 16:03:14.112830 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.112804    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-4wp2q"]
Sep 16 16:03:14.116509 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.116487    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-4wp2q"
Sep 16 16:03:14.118486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.118464    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 16:03:14.118684 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.118474    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-75bhf\""
Sep 16 16:03:14.125979 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.125478    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-4wp2q"]
Sep 16 16:03:14.199899 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.199869    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a194273b-a5f5-4ab2-a47f-3e87dc940b54-cert\") pod \"llmisvc-controller-manager-6c85c996cc-nw4bw\" (UID: \"a194273b-a5f5-4ab2-a47f-3e87dc940b54\") " pod="kserve/llmisvc-controller-manager-6c85c996cc-nw4bw"
Sep 16 16:03:14.200104 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.199911    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9e7515de-5f8e-4760-aee4-20245bfa806e-data\") pod \"seaweedfs-64568bcd49-4wp2q\" (UID: \"9e7515de-5f8e-4760-aee4-20245bfa806e\") " pod="kserve/seaweedfs-64568bcd49-4wp2q"
Sep 16 16:03:14.200104 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.199948    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-497x4\" (UniqueName: \"kubernetes.io/projected/a194273b-a5f5-4ab2-a47f-3e87dc940b54-kube-api-access-497x4\") pod \"llmisvc-controller-manager-6c85c996cc-nw4bw\" (UID: \"a194273b-a5f5-4ab2-a47f-3e87dc940b54\") " pod="kserve/llmisvc-controller-manager-6c85c996cc-nw4bw"
Sep 16 16:03:14.200104 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.199971    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k2kt\" (UniqueName: \"kubernetes.io/projected/9e7515de-5f8e-4760-aee4-20245bfa806e-kube-api-access-5k2kt\") pod \"seaweedfs-64568bcd49-4wp2q\" (UID: \"9e7515de-5f8e-4760-aee4-20245bfa806e\") " pod="kserve/seaweedfs-64568bcd49-4wp2q"
Sep 16 16:03:14.300454 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.300420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a194273b-a5f5-4ab2-a47f-3e87dc940b54-cert\") pod \"llmisvc-controller-manager-6c85c996cc-nw4bw\" (UID: \"a194273b-a5f5-4ab2-a47f-3e87dc940b54\") " pod="kserve/llmisvc-controller-manager-6c85c996cc-nw4bw"
Sep 16 16:03:14.300683 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.300463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9e7515de-5f8e-4760-aee4-20245bfa806e-data\") pod \"seaweedfs-64568bcd49-4wp2q\" (UID: \"9e7515de-5f8e-4760-aee4-20245bfa806e\") " pod="kserve/seaweedfs-64568bcd49-4wp2q"
Sep 16 16:03:14.300683 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.300489    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-497x4\" (UniqueName: \"kubernetes.io/projected/a194273b-a5f5-4ab2-a47f-3e87dc940b54-kube-api-access-497x4\") pod \"llmisvc-controller-manager-6c85c996cc-nw4bw\" (UID: \"a194273b-a5f5-4ab2-a47f-3e87dc940b54\") " pod="kserve/llmisvc-controller-manager-6c85c996cc-nw4bw"
Sep 16 16:03:14.300683 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.300563    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k2kt\" (UniqueName: \"kubernetes.io/projected/9e7515de-5f8e-4760-aee4-20245bfa806e-kube-api-access-5k2kt\") pod \"seaweedfs-64568bcd49-4wp2q\" (UID: \"9e7515de-5f8e-4760-aee4-20245bfa806e\") " pod="kserve/seaweedfs-64568bcd49-4wp2q"
Sep 16 16:03:14.300894 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.300875    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9e7515de-5f8e-4760-aee4-20245bfa806e-data\") pod \"seaweedfs-64568bcd49-4wp2q\" (UID: \"9e7515de-5f8e-4760-aee4-20245bfa806e\") " pod="kserve/seaweedfs-64568bcd49-4wp2q"
Sep 16 16:03:14.302995 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.302974    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a194273b-a5f5-4ab2-a47f-3e87dc940b54-cert\") pod \"llmisvc-controller-manager-6c85c996cc-nw4bw\" (UID: \"a194273b-a5f5-4ab2-a47f-3e87dc940b54\") " pod="kserve/llmisvc-controller-manager-6c85c996cc-nw4bw"
Sep 16 16:03:14.309315 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.309288    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k2kt\" (UniqueName: \"kubernetes.io/projected/9e7515de-5f8e-4760-aee4-20245bfa806e-kube-api-access-5k2kt\") pod \"seaweedfs-64568bcd49-4wp2q\" (UID: \"9e7515de-5f8e-4760-aee4-20245bfa806e\") " pod="kserve/seaweedfs-64568bcd49-4wp2q"
Sep 16 16:03:14.309435 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.309368    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-497x4\" (UniqueName: \"kubernetes.io/projected/a194273b-a5f5-4ab2-a47f-3e87dc940b54-kube-api-access-497x4\") pod \"llmisvc-controller-manager-6c85c996cc-nw4bw\" (UID: \"a194273b-a5f5-4ab2-a47f-3e87dc940b54\") " pod="kserve/llmisvc-controller-manager-6c85c996cc-nw4bw"
Sep 16 16:03:14.405969 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.405935    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6c85c996cc-nw4bw"
Sep 16 16:03:14.429982 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.429948    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-4wp2q"
Sep 16 16:03:14.680104 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.680079    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6c85c996cc-nw4bw"]
Sep 16 16:03:14.682329 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:03:14.682297    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda194273b_a5f5_4ab2_a47f_3e87dc940b54.slice/crio-98249ba4ace6b32b50d111dc82ed9f024f7bd8286043c3271aca6a403302fb4e WatchSource:0}: Error finding container 98249ba4ace6b32b50d111dc82ed9f024f7bd8286043c3271aca6a403302fb4e: Status 404 returned error can't find the container with id 98249ba4ace6b32b50d111dc82ed9f024f7bd8286043c3271aca6a403302fb4e
Sep 16 16:03:14.707881 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:14.707853    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-4wp2q"]
Sep 16 16:03:14.710196 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:03:14.710171    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7515de_5f8e_4760_aee4_20245bfa806e.slice/crio-481ccc5bd3386358bf7c7d08beb826dfab5301e49b641a041c18b21409efad64 WatchSource:0}: Error finding container 481ccc5bd3386358bf7c7d08beb826dfab5301e49b641a041c18b21409efad64: Status 404 returned error can't find the container with id 481ccc5bd3386358bf7c7d08beb826dfab5301e49b641a041c18b21409efad64
Sep 16 16:03:15.404307 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:15.404272    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6c85c996cc-nw4bw" event={"ID":"a194273b-a5f5-4ab2-a47f-3e87dc940b54","Type":"ContainerStarted","Data":"98249ba4ace6b32b50d111dc82ed9f024f7bd8286043c3271aca6a403302fb4e"}
Sep 16 16:03:15.405465 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:15.405443    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-4wp2q" event={"ID":"9e7515de-5f8e-4760-aee4-20245bfa806e","Type":"ContainerStarted","Data":"481ccc5bd3386358bf7c7d08beb826dfab5301e49b641a041c18b21409efad64"}
Sep 16 16:03:19.429010 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:19.428976    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6c85c996cc-nw4bw" event={"ID":"a194273b-a5f5-4ab2-a47f-3e87dc940b54","Type":"ContainerStarted","Data":"814bdf8b604e4fe84c75d5db0990b8e58a89d308deac4b60eb98230641d42fcb"}
Sep 16 16:03:19.429470 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:19.429078    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6c85c996cc-nw4bw"
Sep 16 16:03:19.430990 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:19.430961    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-4wp2q" event={"ID":"9e7515de-5f8e-4760-aee4-20245bfa806e","Type":"ContainerStarted","Data":"244950aa8c9a4633bcf5e48e680492c99d4bbd5c1bc7508145956927b571fee0"}
Sep 16 16:03:19.431122 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:19.431106    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-4wp2q"
Sep 16 16:03:19.445174 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:19.445119    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6c85c996cc-nw4bw" podStartSLOduration=1.519456787 podStartE2EDuration="5.445105041s" totalImagesPullingTime="3.925648254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:03:19.443439646 +0000 UTC m=+795.963062423" watchObservedRunningTime="2026-09-16 16:03:19.445105041 +0000 UTC m=+795.964727818"
Sep 16 16:03:19.458481 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:19.458433    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-4wp2q" podStartSLOduration=1.468379445 podStartE2EDuration="5.458419313s" totalImagesPullingTime="3.990039868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:03:19.457437211 +0000 UTC m=+795.977059988" watchObservedRunningTime="2026-09-16 16:03:19.458419313 +0000 UTC m=+795.978042089"
Sep 16 16:03:25.437597 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:25.437569    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-4wp2q"
Sep 16 16:03:50.437980 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:03:50.437942    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6c85c996cc-nw4bw"
Sep 16 16:04:34.314214 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:34.314177    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-wx6zr"]
Sep 16 16:04:34.317998 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:34.317969    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-wx6zr"
Sep 16 16:04:34.320023 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:34.319995    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 16:04:34.320186 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:34.320056    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-z7b58\""
Sep 16 16:04:34.328395 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:34.328029    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-wx6zr"]
Sep 16 16:04:34.341463 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:34.341417    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9-tls-certs\") pod \"model-serving-api-7b954ffc95-wx6zr\" (UID: \"39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9\") " pod="kserve/model-serving-api-7b954ffc95-wx6zr"
Sep 16 16:04:34.341691 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:34.341575    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trh4m\" (UniqueName: \"kubernetes.io/projected/39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9-kube-api-access-trh4m\") pod \"model-serving-api-7b954ffc95-wx6zr\" (UID: \"39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9\") " pod="kserve/model-serving-api-7b954ffc95-wx6zr"
Sep 16 16:04:34.442683 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:34.442496    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9-tls-certs\") pod \"model-serving-api-7b954ffc95-wx6zr\" (UID: \"39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9\") " pod="kserve/model-serving-api-7b954ffc95-wx6zr"
Sep 16 16:04:34.442889 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:34.442798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trh4m\" (UniqueName: \"kubernetes.io/projected/39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9-kube-api-access-trh4m\") pod \"model-serving-api-7b954ffc95-wx6zr\" (UID: \"39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9\") " pod="kserve/model-serving-api-7b954ffc95-wx6zr"
Sep 16 16:04:34.445278 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:34.445253    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9-tls-certs\") pod \"model-serving-api-7b954ffc95-wx6zr\" (UID: \"39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9\") " pod="kserve/model-serving-api-7b954ffc95-wx6zr"
Sep 16 16:04:34.450952 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:34.450928    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trh4m\" (UniqueName: \"kubernetes.io/projected/39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9-kube-api-access-trh4m\") pod \"model-serving-api-7b954ffc95-wx6zr\" (UID: \"39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9\") " pod="kserve/model-serving-api-7b954ffc95-wx6zr"
Sep 16 16:04:34.632556 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:34.632465    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-wx6zr"
Sep 16 16:04:34.911413 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:34.911385    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-wx6zr"]
Sep 16 16:04:34.913635 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:04:34.913605    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d33713_d0a3_4ea9_b6aa_273f0ee0d9f9.slice/crio-1a225bc45ce97af38d29b981c70533db7d03ba941be909df0a93b03b2d88b74e WatchSource:0}: Error finding container 1a225bc45ce97af38d29b981c70533db7d03ba941be909df0a93b03b2d88b74e: Status 404 returned error can't find the container with id 1a225bc45ce97af38d29b981c70533db7d03ba941be909df0a93b03b2d88b74e
Sep 16 16:04:35.802260 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:35.802214    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-wx6zr" event={"ID":"39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9","Type":"ContainerStarted","Data":"1a225bc45ce97af38d29b981c70533db7d03ba941be909df0a93b03b2d88b74e"}
Sep 16 16:04:37.813661 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:37.813628    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-wx6zr" event={"ID":"39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9","Type":"ContainerStarted","Data":"d5b355e4466fe562e27164efd5b6426dc6ee76b0565e62b422bf5c4b29b90fa3"}
Sep 16 16:04:37.814065 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:37.813710    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-wx6zr"
Sep 16 16:04:37.832336 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:37.832282    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-wx6zr" podStartSLOduration=1.532840276 podStartE2EDuration="3.832265681s" totalImagesPullingTime="2.299425405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:04:37.830428889 +0000 UTC m=+874.350051677" watchObservedRunningTime="2026-09-16 16:04:37.832265681 +0000 UTC m=+874.351888457"
Sep 16 16:04:48.822779 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:04:48.822749    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-wx6zr"
Sep 16 16:06:29.416640 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.416605    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 16:06:29.420268 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.420247    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.422560 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.422498    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 16:06:29.422678 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.422504    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 16:06:29.423081 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.423061    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 16:06:29.423795 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.423589    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-j795p\""
Sep 16 16:06:29.423795 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.423098    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 16:06:29.423795 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.423138    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-mfjsx\""
Sep 16 16:06:29.423795 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.423137    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:06:29.430459 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.430438    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 16:06:29.589949 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.589911    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b61ec23c-3b87-4f0e-930e-36f6a46fc892-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.589949 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.589951    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2psv9\" (UniqueName: \"kubernetes.io/projected/b61ec23c-3b87-4f0e-930e-36f6a46fc892-kube-api-access-2psv9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.590171 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.589977    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.590171 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.590061    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.590171 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.590119    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.590171 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.590148    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.590319 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.590189    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.590319 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.590267    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ec23c-3b87-4f0e-930e-36f6a46fc892-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.691084 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.691003    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b61ec23c-3b87-4f0e-930e-36f6a46fc892-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.691084 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.691040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2psv9\" (UniqueName: \"kubernetes.io/projected/b61ec23c-3b87-4f0e-930e-36f6a46fc892-kube-api-access-2psv9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.691084 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.691068    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.691343 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.691091    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.691343 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.691132    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.691343 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.691149    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.691343 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.691188    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.691343 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.691236    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ec23c-3b87-4f0e-930e-36f6a46fc892-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.691620 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.691470    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.691620 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.691567    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.691722 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.691670    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.691722 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.691698    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.691722 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.691712    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b61ec23c-3b87-4f0e-930e-36f6a46fc892-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.693453 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.693428    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.693669 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.693651    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ec23c-3b87-4f0e-930e-36f6a46fc892-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.699800 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.699773    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2psv9\" (UniqueName: \"kubernetes.io/projected/b61ec23c-3b87-4f0e-930e-36f6a46fc892-kube-api-access-2psv9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:29.731843 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:29.731813    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:06:30.007172 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:30.007143    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 16:06:30.009571 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:06:30.009537    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61ec23c_3b87_4f0e_930e_36f6a46fc892.slice/crio-6dce7efe491ff266361a3cbfa7cd425f34b898b961a7df54e4b65f966ba93fa0 WatchSource:0}: Error finding container 6dce7efe491ff266361a3cbfa7cd425f34b898b961a7df54e4b65f966ba93fa0: Status 404 returned error can't find the container with id 6dce7efe491ff266361a3cbfa7cd425f34b898b961a7df54e4b65f966ba93fa0
Sep 16 16:06:30.011864 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:30.011848    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 16:06:30.348926 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:30.348892    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b61ec23c-3b87-4f0e-930e-36f6a46fc892","Type":"ContainerStarted","Data":"6dce7efe491ff266361a3cbfa7cd425f34b898b961a7df54e4b65f966ba93fa0"}
Sep 16 16:06:34.374902 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:34.374864    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b61ec23c-3b87-4f0e-930e-36f6a46fc892","Type":"ContainerStarted","Data":"97228be209675d307e94e0c5695584a27803b2c48cb262cf1b2833b9df333f85"}
Sep 16 16:06:37.392682 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:37.392649    2574 generic.go:356] "Generic (PLEG): container finished" podID="b61ec23c-3b87-4f0e-930e-36f6a46fc892" containerID="97228be209675d307e94e0c5695584a27803b2c48cb262cf1b2833b9df333f85" exitCode=0
Sep 16 16:06:37.393069 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:37.392696    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b61ec23c-3b87-4f0e-930e-36f6a46fc892","Type":"ContainerDied","Data":"97228be209675d307e94e0c5695584a27803b2c48cb262cf1b2833b9df333f85"}
Sep 16 16:06:41.431224 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:41.431174    2574 generic.go:356] "Generic (PLEG): container finished" podID="f9980823-1d5e-4996-8eec-7054d9091b82" containerID="a219d90dc0887343cec8a339120a5847ddb094f16ac55b6c17f1457d70bbdeef" exitCode=0
Sep 16 16:06:41.431753 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:41.431257    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c" event={"ID":"f9980823-1d5e-4996-8eec-7054d9091b82","Type":"ContainerDied","Data":"a219d90dc0887343cec8a339120a5847ddb094f16ac55b6c17f1457d70bbdeef"}
Sep 16 16:06:41.431933 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:41.431912    2574 scope.go:117] "RemoveContainer" containerID="a219d90dc0887343cec8a339120a5847ddb094f16ac55b6c17f1457d70bbdeef"
Sep 16 16:06:41.433342 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:41.433151    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 16:06:41.433342 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:41.433243    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 16:06:41.433342 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:41.433289    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 16:06:42.442045 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:42.441964    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c" event={"ID":"f9980823-1d5e-4996-8eec-7054d9091b82","Type":"ContainerStarted","Data":"e14b2f7395b55d4bd467409db63202abdef17ab8f5d704b79904ac2ce7a7be08"}
Sep 16 16:06:42.741218 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:42.741031    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:06:42.742372 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:42.742341    2574 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9582c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Sep 16 16:06:42.742537 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:42.742401    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c" podUID="f9980823-1d5e-4996-8eec-7054d9091b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Sep 16 16:06:42.750096 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:42.750033    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:06:43.741337 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:43.741297    2574 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9582c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Sep 16 16:06:43.741774 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:43.741384    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c" podUID="f9980823-1d5e-4996-8eec-7054d9091b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Sep 16 16:06:44.741242 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:44.741192    2574 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9582c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Sep 16 16:06:44.741242 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:44.741247    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c" podUID="f9980823-1d5e-4996-8eec-7054d9091b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Sep 16 16:06:45.742505 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:45.742144    2574 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9582c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Sep 16 16:06:45.742505 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:45.742211    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c" podUID="f9980823-1d5e-4996-8eec-7054d9091b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Sep 16 16:06:46.741298 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:46.741253    2574 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9582c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Sep 16 16:06:46.741546 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:46.741332    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c" podUID="f9980823-1d5e-4996-8eec-7054d9091b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Sep 16 16:06:48.346637 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:48.346063    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:06:48.348624 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:06:48.348569    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9582c"
Sep 16 16:07:09.597137 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:07:09.597097    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b61ec23c-3b87-4f0e-930e-36f6a46fc892","Type":"ContainerStarted","Data":"e05ec641036c378c7130b9c7bfddf9b4957d0929263848284fd60824b72849c6"}
Sep 16 16:07:09.618305 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:07:09.618248    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.4416681479999998 podStartE2EDuration="40.618231115s" totalImagesPullingTime="35.589493994s" totalInitContainerRuntime="2.587068973s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:06:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:07:09.616956114 +0000 UTC m=+1026.136578890" watchObservedRunningTime="2026-09-16 16:07:09.618231115 +0000 UTC m=+1026.137853892"
Sep 16 16:08:27.846129 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:27.846083    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"]
Sep 16 16:08:27.850895 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:27.850866    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"
Sep 16 16:08:27.853285 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:27.853258    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 16 16:08:27.854246 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:27.854227    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-wpvzp\""
Sep 16 16:08:27.854352 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:27.854275    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 16 16:08:27.854352 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:27.854232    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 16 16:08:27.854440 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:27.854379    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-m8xfs\""
Sep 16 16:08:27.857823 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:27.857802    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"]
Sep 16 16:08:27.973692 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:27.973654    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f8e7fd-3134-4af2-a4d7-fc231c32b436-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k\" (UID: \"54f8e7fd-3134-4af2-a4d7-fc231c32b436\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"
Sep 16 16:08:27.973885 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:27.973715    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsmlg\" (UniqueName: \"kubernetes.io/projected/54f8e7fd-3134-4af2-a4d7-fc231c32b436-kube-api-access-nsmlg\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k\" (UID: \"54f8e7fd-3134-4af2-a4d7-fc231c32b436\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"
Sep 16 16:08:28.075027 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:28.074985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsmlg\" (UniqueName: \"kubernetes.io/projected/54f8e7fd-3134-4af2-a4d7-fc231c32b436-kube-api-access-nsmlg\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k\" (UID: \"54f8e7fd-3134-4af2-a4d7-fc231c32b436\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"
Sep 16 16:08:28.075206 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:28.075135    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f8e7fd-3134-4af2-a4d7-fc231c32b436-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k\" (UID: \"54f8e7fd-3134-4af2-a4d7-fc231c32b436\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"
Sep 16 16:08:28.077849 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:28.077820    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f8e7fd-3134-4af2-a4d7-fc231c32b436-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k\" (UID: \"54f8e7fd-3134-4af2-a4d7-fc231c32b436\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"
Sep 16 16:08:28.083597 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:28.083575    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsmlg\" (UniqueName: \"kubernetes.io/projected/54f8e7fd-3134-4af2-a4d7-fc231c32b436-kube-api-access-nsmlg\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k\" (UID: \"54f8e7fd-3134-4af2-a4d7-fc231c32b436\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"
Sep 16 16:08:28.163586 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:28.163447    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"
Sep 16 16:08:28.438490 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:28.438453    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"]
Sep 16 16:08:28.440343 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:08:28.440299    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f8e7fd_3134_4af2_a4d7_fc231c32b436.slice/crio-fba630023020ee3e50a6bf62b64ea0c934ce370612cbe8d7259c3fc32b7bc8e2 WatchSource:0}: Error finding container fba630023020ee3e50a6bf62b64ea0c934ce370612cbe8d7259c3fc32b7bc8e2: Status 404 returned error can't find the container with id fba630023020ee3e50a6bf62b64ea0c934ce370612cbe8d7259c3fc32b7bc8e2
Sep 16 16:08:28.999196 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:28.999154    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k" event={"ID":"54f8e7fd-3134-4af2-a4d7-fc231c32b436","Type":"ContainerStarted","Data":"fba630023020ee3e50a6bf62b64ea0c934ce370612cbe8d7259c3fc32b7bc8e2"}
Sep 16 16:08:31.013055 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:31.013016    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k" event={"ID":"54f8e7fd-3134-4af2-a4d7-fc231c32b436","Type":"ContainerStarted","Data":"9caceb3ac5acf2a58892dab26d12ea57030c7324cb942b4df24da727fc48653d"}
Sep 16 16:08:31.013584 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:31.013142    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"
Sep 16 16:08:31.028883 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:08:31.028834    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k" podStartSLOduration=1.550573867 podStartE2EDuration="4.028815052s" totalImagesPullingTime="2.478241185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:08:31.028025109 +0000 UTC m=+1107.547647888" watchObservedRunningTime="2026-09-16 16:08:31.028815052 +0000 UTC m=+1107.548437829"
Sep 16 16:09:02.019872 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:02.019838    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"
Sep 16 16:09:04.511021 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:04.510981    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 16:09:08.484575 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:08.484541    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"]
Sep 16 16:09:12.020364 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:12.020315    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k" podUID="54f8e7fd-3134-4af2-a4d7-fc231c32b436" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:09:19.533597 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:19.533530    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="b61ec23c-3b87-4f0e-930e-36f6a46fc892" containerName="main" containerID="cri-o://e05ec641036c378c7130b9c7bfddf9b4957d0929263848284fd60824b72849c6" gracePeriod=45
Sep 16 16:09:20.264391 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.264362    2574 generic.go:356] "Generic (PLEG): container finished" podID="b61ec23c-3b87-4f0e-930e-36f6a46fc892" containerID="e05ec641036c378c7130b9c7bfddf9b4957d0929263848284fd60824b72849c6" exitCode=0
Sep 16 16:09:20.264508 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.264410    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b61ec23c-3b87-4f0e-930e-36f6a46fc892","Type":"ContainerDied","Data":"e05ec641036c378c7130b9c7bfddf9b4957d0929263848284fd60824b72849c6"}
Sep 16 16:09:20.284238 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.284216    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:09:20.368688 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.368649    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-tmp-dir\") pod \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") "
Sep 16 16:09:20.368853 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.368720    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ec23c-3b87-4f0e-930e-36f6a46fc892-tls-certs\") pod \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") "
Sep 16 16:09:20.368853 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.368751    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2psv9\" (UniqueName: \"kubernetes.io/projected/b61ec23c-3b87-4f0e-930e-36f6a46fc892-kube-api-access-2psv9\") pod \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") "
Sep 16 16:09:20.368853 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.368768    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-dshm\") pod \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") "
Sep 16 16:09:20.368853 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.368787    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b61ec23c-3b87-4f0e-930e-36f6a46fc892-cabundle-cert\") pod \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") "
Sep 16 16:09:20.368853 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.368805    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-kserve-provision-location\") pod \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") "
Sep 16 16:09:20.369063 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.368878    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-model-cache\") pod \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") "
Sep 16 16:09:20.369063 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.368923    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-home\") pod \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\" (UID: \"b61ec23c-3b87-4f0e-930e-36f6a46fc892\") "
Sep 16 16:09:20.369204 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.369177    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61ec23c-3b87-4f0e-930e-36f6a46fc892-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b61ec23c-3b87-4f0e-930e-36f6a46fc892" (UID: "b61ec23c-3b87-4f0e-930e-36f6a46fc892"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:09:20.369456 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.369428    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-model-cache" (OuterVolumeSpecName: "model-cache") pod "b61ec23c-3b87-4f0e-930e-36f6a46fc892" (UID: "b61ec23c-3b87-4f0e-930e-36f6a46fc892"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:09:20.369841 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.369813    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-home" (OuterVolumeSpecName: "home") pod "b61ec23c-3b87-4f0e-930e-36f6a46fc892" (UID: "b61ec23c-3b87-4f0e-930e-36f6a46fc892"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:09:20.371093 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.371063    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61ec23c-3b87-4f0e-930e-36f6a46fc892-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b61ec23c-3b87-4f0e-930e-36f6a46fc892" (UID: "b61ec23c-3b87-4f0e-930e-36f6a46fc892"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:09:20.371197 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.371097    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61ec23c-3b87-4f0e-930e-36f6a46fc892-kube-api-access-2psv9" (OuterVolumeSpecName: "kube-api-access-2psv9") pod "b61ec23c-3b87-4f0e-930e-36f6a46fc892" (UID: "b61ec23c-3b87-4f0e-930e-36f6a46fc892"). InnerVolumeSpecName "kube-api-access-2psv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:09:20.371197 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.371126    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-dshm" (OuterVolumeSpecName: "dshm") pod "b61ec23c-3b87-4f0e-930e-36f6a46fc892" (UID: "b61ec23c-3b87-4f0e-930e-36f6a46fc892"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:09:20.386917 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.386881    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b61ec23c-3b87-4f0e-930e-36f6a46fc892" (UID: "b61ec23c-3b87-4f0e-930e-36f6a46fc892"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:09:20.390045 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.390012    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b61ec23c-3b87-4f0e-930e-36f6a46fc892" (UID: "b61ec23c-3b87-4f0e-930e-36f6a46fc892"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:09:20.470449 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.470409    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61ec23c-3b87-4f0e-930e-36f6a46fc892-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:09:20.470449 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.470445    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2psv9\" (UniqueName: \"kubernetes.io/projected/b61ec23c-3b87-4f0e-930e-36f6a46fc892-kube-api-access-2psv9\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:09:20.470449 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.470458    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:09:20.470736 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.470470    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b61ec23c-3b87-4f0e-930e-36f6a46fc892-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:09:20.470736 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.470483    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:09:20.470736 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.470493    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:09:20.470736 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.470503    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:09:20.470736 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:20.470542    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61ec23c-3b87-4f0e-930e-36f6a46fc892-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:09:21.271209 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:21.271170    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b61ec23c-3b87-4f0e-930e-36f6a46fc892","Type":"ContainerDied","Data":"6dce7efe491ff266361a3cbfa7cd425f34b898b961a7df54e4b65f966ba93fa0"}
Sep 16 16:09:21.271209 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:21.271204    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 16:09:21.271698 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:21.271269    2574 scope.go:117] "RemoveContainer" containerID="e05ec641036c378c7130b9c7bfddf9b4957d0929263848284fd60824b72849c6"
Sep 16 16:09:21.281101 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:21.281083    2574 scope.go:117] "RemoveContainer" containerID="97228be209675d307e94e0c5695584a27803b2c48cb262cf1b2833b9df333f85"
Sep 16 16:09:21.293864 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:21.293835    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 16:09:21.299489 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:21.299462    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 16:09:21.994401 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:21.994360    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b61ec23c-3b87-4f0e-930e-36f6a46fc892" path="/var/lib/kubelet/pods/b61ec23c-3b87-4f0e-930e-36f6a46fc892/volumes"
Sep 16 16:09:22.019741 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:22.019697    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k" podUID="54f8e7fd-3134-4af2-a4d7-fc231c32b436" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:09:23.505185 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:23.505147    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k" podUID="54f8e7fd-3134-4af2-a4d7-fc231c32b436" containerName="main" containerID="cri-o://9caceb3ac5acf2a58892dab26d12ea57030c7324cb942b4df24da727fc48653d" gracePeriod=45
Sep 16 16:09:32.020338 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:32.020242    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k" podUID="54f8e7fd-3134-4af2-a4d7-fc231c32b436" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:09:32.020807 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:32.020368    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"
Sep 16 16:09:42.019773 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:42.019685    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k" podUID="54f8e7fd-3134-4af2-a4d7-fc231c32b436" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:09:52.019913 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:52.019874    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k" podUID="54f8e7fd-3134-4af2-a4d7-fc231c32b436" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:09:53.750591 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:53.750565    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"
Sep 16 16:09:53.768681 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:53.768609    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f8e7fd-3134-4af2-a4d7-fc231c32b436-tls-certs\") pod \"54f8e7fd-3134-4af2-a4d7-fc231c32b436\" (UID: \"54f8e7fd-3134-4af2-a4d7-fc231c32b436\") "
Sep 16 16:09:53.768681 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:53.768663    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsmlg\" (UniqueName: \"kubernetes.io/projected/54f8e7fd-3134-4af2-a4d7-fc231c32b436-kube-api-access-nsmlg\") pod \"54f8e7fd-3134-4af2-a4d7-fc231c32b436\" (UID: \"54f8e7fd-3134-4af2-a4d7-fc231c32b436\") "
Sep 16 16:09:53.770813 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:53.770783    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f8e7fd-3134-4af2-a4d7-fc231c32b436-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54f8e7fd-3134-4af2-a4d7-fc231c32b436" (UID: "54f8e7fd-3134-4af2-a4d7-fc231c32b436"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:09:53.770957 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:53.770939    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f8e7fd-3134-4af2-a4d7-fc231c32b436-kube-api-access-nsmlg" (OuterVolumeSpecName: "kube-api-access-nsmlg") pod "54f8e7fd-3134-4af2-a4d7-fc231c32b436" (UID: "54f8e7fd-3134-4af2-a4d7-fc231c32b436"). InnerVolumeSpecName "kube-api-access-nsmlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:09:53.869915 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:53.869867    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f8e7fd-3134-4af2-a4d7-fc231c32b436-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:09:53.869915 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:53.869909    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsmlg\" (UniqueName: \"kubernetes.io/projected/54f8e7fd-3134-4af2-a4d7-fc231c32b436-kube-api-access-nsmlg\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:09:54.447101 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:54.447072    2574 generic.go:356] "Generic (PLEG): container finished" podID="54f8e7fd-3134-4af2-a4d7-fc231c32b436" containerID="9caceb3ac5acf2a58892dab26d12ea57030c7324cb942b4df24da727fc48653d" exitCode=0
Sep 16 16:09:54.447278 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:54.447109    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"
Sep 16 16:09:54.447278 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:54.447142    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k" event={"ID":"54f8e7fd-3134-4af2-a4d7-fc231c32b436","Type":"ContainerDied","Data":"9caceb3ac5acf2a58892dab26d12ea57030c7324cb942b4df24da727fc48653d"}
Sep 16 16:09:54.447278 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:54.447195    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k" event={"ID":"54f8e7fd-3134-4af2-a4d7-fc231c32b436","Type":"ContainerDied","Data":"fba630023020ee3e50a6bf62b64ea0c934ce370612cbe8d7259c3fc32b7bc8e2"}
Sep 16 16:09:54.447278 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:54.447218    2574 scope.go:117] "RemoveContainer" containerID="9caceb3ac5acf2a58892dab26d12ea57030c7324cb942b4df24da727fc48653d"
Sep 16 16:09:54.455222 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:54.455205    2574 scope.go:117] "RemoveContainer" containerID="9caceb3ac5acf2a58892dab26d12ea57030c7324cb942b4df24da727fc48653d"
Sep 16 16:09:54.455508 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:09:54.455483    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9caceb3ac5acf2a58892dab26d12ea57030c7324cb942b4df24da727fc48653d\": container with ID starting with 9caceb3ac5acf2a58892dab26d12ea57030c7324cb942b4df24da727fc48653d not found: ID does not exist" containerID="9caceb3ac5acf2a58892dab26d12ea57030c7324cb942b4df24da727fc48653d"
Sep 16 16:09:54.455605 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:54.455509    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9caceb3ac5acf2a58892dab26d12ea57030c7324cb942b4df24da727fc48653d"} err="failed to get container status \"9caceb3ac5acf2a58892dab26d12ea57030c7324cb942b4df24da727fc48653d\": rpc error: code = NotFound desc = could not find container \"9caceb3ac5acf2a58892dab26d12ea57030c7324cb942b4df24da727fc48653d\": container with ID starting with 9caceb3ac5acf2a58892dab26d12ea57030c7324cb942b4df24da727fc48653d not found: ID does not exist"
Sep 16 16:09:54.463837 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:54.463815    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"]
Sep 16 16:09:54.467055 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:54.467035    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-8cz2k"]
Sep 16 16:09:55.994032 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:55.994001    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54f8e7fd-3134-4af2-a4d7-fc231c32b436" path="/var/lib/kubelet/pods/54f8e7fd-3134-4af2-a4d7-fc231c32b436/volumes"
Sep 16 16:09:58.635706 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.635678    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"]
Sep 16 16:09:58.636245 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.636225    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61ec23c-3b87-4f0e-930e-36f6a46fc892" containerName="main"
Sep 16 16:09:58.636323 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.636247    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61ec23c-3b87-4f0e-930e-36f6a46fc892" containerName="main"
Sep 16 16:09:58.636323 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.636273    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54f8e7fd-3134-4af2-a4d7-fc231c32b436" containerName="main"
Sep 16 16:09:58.636323 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.636281    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="54f8e7fd-3134-4af2-a4d7-fc231c32b436" containerName="main"
Sep 16 16:09:58.636323 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.636300    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61ec23c-3b87-4f0e-930e-36f6a46fc892" containerName="storage-initializer"
Sep 16 16:09:58.636323 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.636310    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61ec23c-3b87-4f0e-930e-36f6a46fc892" containerName="storage-initializer"
Sep 16 16:09:58.636579 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.636415    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b61ec23c-3b87-4f0e-930e-36f6a46fc892" containerName="main"
Sep 16 16:09:58.636579 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.636428    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="54f8e7fd-3134-4af2-a4d7-fc231c32b436" containerName="main"
Sep 16 16:09:58.641059 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.641039    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.643843 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.643814    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 16 16:09:58.643965 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.643874    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-sbv69\""
Sep 16 16:09:58.643965 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.643894    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 16:09:58.643965 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.643939    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 16:09:58.662652 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.662627    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"]
Sep 16 16:09:58.711641 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.711599    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd75437-1890-4a13-88ec-5bb33d22a927-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.711641 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.711637    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.711826 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.711654    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.711826 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.711704    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djqkh\" (UniqueName: \"kubernetes.io/projected/fdd75437-1890-4a13-88ec-5bb33d22a927-kube-api-access-djqkh\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.711826 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.711748    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.711826 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.711779    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.711826 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.711793    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.711986 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.711893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.711986 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.711908    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.812514 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.812474    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.812514 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.812535    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.812793 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.812587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd75437-1890-4a13-88ec-5bb33d22a927-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.812793 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.812612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.812793 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.812636    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.812793 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.812664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djqkh\" (UniqueName: \"kubernetes.io/projected/fdd75437-1890-4a13-88ec-5bb33d22a927-kube-api-access-djqkh\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.812793 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.812700    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.812793 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.812739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.812793 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.812764    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.813114 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.813004    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.813360 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.813288    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.813360 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.813331    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.813360 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.813331    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd75437-1890-4a13-88ec-5bb33d22a927-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.813626 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.813409    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.815168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.815149    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.815372 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.815353    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.826902 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.826883    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.837547 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.837505    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djqkh\" (UniqueName: \"kubernetes.io/projected/fdd75437-1890-4a13-88ec-5bb33d22a927-kube-api-access-djqkh\") pod \"router-gateway-1-openshift-default-5b4d9458bd-6q9l4\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:58.952110 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:58.952014    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:09:59.229060 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:59.228978    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"]
Sep 16 16:09:59.232039 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:09:59.232014    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd75437_1890_4a13_88ec_5bb33d22a927.slice/crio-dbe0418af9fd5de9454b97ddc4f2dd6817c97d8bec77a36cfefa4fe53768b557 WatchSource:0}: Error finding container dbe0418af9fd5de9454b97ddc4f2dd6817c97d8bec77a36cfefa4fe53768b557: Status 404 returned error can't find the container with id dbe0418af9fd5de9454b97ddc4f2dd6817c97d8bec77a36cfefa4fe53768b557
Sep 16 16:09:59.234013 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:59.233977    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 16:09:59.234092 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:59.234067    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 16:09:59.234137 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:59.234124    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 16:09:59.473791 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:59.473747    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4" event={"ID":"fdd75437-1890-4a13-88ec-5bb33d22a927","Type":"ContainerStarted","Data":"8db9ba04e37ee844aff750b7ed3ff2536d24f9c754bbba5188f9351afccaf76a"}
Sep 16 16:09:59.473980 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:59.473805    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4" event={"ID":"fdd75437-1890-4a13-88ec-5bb33d22a927","Type":"ContainerStarted","Data":"dbe0418af9fd5de9454b97ddc4f2dd6817c97d8bec77a36cfefa4fe53768b557"}
Sep 16 16:09:59.495018 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:59.494925    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4" podStartSLOduration=1.494909557 podStartE2EDuration="1.494909557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:09:59.493287239 +0000 UTC m=+1196.012910050" watchObservedRunningTime="2026-09-16 16:09:59.494909557 +0000 UTC m=+1196.014532333"
Sep 16 16:09:59.952640 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:09:59.952604    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:10:00.957156 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:00.957126    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:10:01.484582 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:01.484550    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:10:01.485434 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:01.485415    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:10:04.961606 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:04.961569    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"]
Sep 16 16:10:04.966480 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:04.966462    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:04.969312 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:04.969288    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-srfpt\""
Sep 16 16:10:04.969427 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:04.969315    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 16 16:10:04.973235 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:04.973211    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"]
Sep 16 16:10:05.074669 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.074631    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw7fz\" (UniqueName: \"kubernetes.io/projected/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-kube-api-access-xw7fz\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.074869 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.074695    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.074869 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.074815    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.075090 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.075067    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.075167 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.075126    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.075167 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.075154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.176062 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.176027    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.176062 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.176067    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.176317 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.176088    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.176317 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.176260    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw7fz\" (UniqueName: \"kubernetes.io/projected/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-kube-api-access-xw7fz\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.176433 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.176356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.176433 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.176393    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.176548 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.176533    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.176614 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.176554    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.176839 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.176812    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.178791 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.178762    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.179173 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.179153    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.184092 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.184061    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw7fz\" (UniqueName: \"kubernetes.io/projected/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-kube-api-access-xw7fz\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.277096 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.276999    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:05.549132 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:05.548947    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"]
Sep 16 16:10:05.551374 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:10:05.551347    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53b82f3_57d1_4af5_ad9e_9e52c6ae499c.slice/crio-2fd2a40106e28c5674c21d5ccdd228edece4ba86bdabafe0410b0a55a958fc66 WatchSource:0}: Error finding container 2fd2a40106e28c5674c21d5ccdd228edece4ba86bdabafe0410b0a55a958fc66: Status 404 returned error can't find the container with id 2fd2a40106e28c5674c21d5ccdd228edece4ba86bdabafe0410b0a55a958fc66
Sep 16 16:10:06.530540 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:06.529718    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b" event={"ID":"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c","Type":"ContainerStarted","Data":"2fd2a40106e28c5674c21d5ccdd228edece4ba86bdabafe0410b0a55a958fc66"}
Sep 16 16:10:07.637544 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.637198    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"]
Sep 16 16:10:07.641018 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.640980    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.644973 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.644949    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 16:10:07.645177 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.645160    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-n52tb\""
Sep 16 16:10:07.645309 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.645290    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 16:10:07.645429 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.645407    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 16 16:10:07.664497 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.664466    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"]
Sep 16 16:10:07.703486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.703420    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.703486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.703460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.703486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.703487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmj98\" (UniqueName: \"kubernetes.io/projected/7e48a76b-6419-4df7-9484-7bf5271c61bc-kube-api-access-mmj98\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.703774 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.703606    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e48a76b-6419-4df7-9484-7bf5271c61bc-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.703774 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.703654    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.703774 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.703702    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.703774 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.703728    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.703987 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.703829    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.703987 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.703868    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.805435 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.805349    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.805435 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.805391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.805435 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.805417    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmj98\" (UniqueName: \"kubernetes.io/projected/7e48a76b-6419-4df7-9484-7bf5271c61bc-kube-api-access-mmj98\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.805732 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.805455    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e48a76b-6419-4df7-9484-7bf5271c61bc-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.805732 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.805485    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.805732 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.805504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.805732 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.805559    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.805732 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.805623    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.805732 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.805650    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.806031 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.805794    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.806031 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.805898    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.806109 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.806035    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.806193 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.806167    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e48a76b-6419-4df7-9484-7bf5271c61bc-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.806297 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.806272    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.811053 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.811024    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.821052 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.820998    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.838143 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.838106    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.845574 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.845548    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmj98\" (UniqueName: \"kubernetes.io/projected/7e48a76b-6419-4df7-9484-7bf5271c61bc-kube-api-access-mmj98\") pod \"router-gateway-1-openshift-default-66bbccc5fb-jcfqf\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:07.951848 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:07.951805    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:08.089773 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:08.089693    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"]
Sep 16 16:10:08.230253 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:08.230213    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"]
Sep 16 16:10:08.238186 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:10:08.238160    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e48a76b_6419_4df7_9484_7bf5271c61bc.slice/crio-7abff80ce99c9c64566d8d68fb61cca6f05bfd522ae6a2791d4a2ff59c13105c WatchSource:0}: Error finding container 7abff80ce99c9c64566d8d68fb61cca6f05bfd522ae6a2791d4a2ff59c13105c: Status 404 returned error can't find the container with id 7abff80ce99c9c64566d8d68fb61cca6f05bfd522ae6a2791d4a2ff59c13105c
Sep 16 16:10:08.240192 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:08.240162    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 16:10:08.240243 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:08.240227    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 16:10:08.240278 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:08.240258    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 16:10:08.545304 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:08.545264    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf" event={"ID":"7e48a76b-6419-4df7-9484-7bf5271c61bc","Type":"ContainerStarted","Data":"e70e51830b9c0aad20eb372d3fe961b0bb4403a3b041fb8c44357f579505ea02"}
Sep 16 16:10:08.545461 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:08.545331    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf" event={"ID":"7e48a76b-6419-4df7-9484-7bf5271c61bc","Type":"ContainerStarted","Data":"7abff80ce99c9c64566d8d68fb61cca6f05bfd522ae6a2791d4a2ff59c13105c"}
Sep 16 16:10:08.546889 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:08.546850    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b" event={"ID":"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c","Type":"ContainerStarted","Data":"7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28"}
Sep 16 16:10:08.566916 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:08.566632    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf" podStartSLOduration=1.566614183 podStartE2EDuration="1.566614183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:10:08.56590136 +0000 UTC m=+1205.085524136" watchObservedRunningTime="2026-09-16 16:10:08.566614183 +0000 UTC m=+1205.086236959"
Sep 16 16:10:08.584100 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:08.584054    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b" podStartSLOduration=2.631910291 podStartE2EDuration="4.584037006s" totalImagesPullingTime="1.952126715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:10:08.583046082 +0000 UTC m=+1205.102668899" watchObservedRunningTime="2026-09-16 16:10:08.584037006 +0000 UTC m=+1205.103659781"
Sep 16 16:10:08.633549 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:10:08.633497    2574 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Sep 16 16:10:08.633721 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:10:08.633596    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tls-certs podName:c53b82f3-57d1-4af5-ad9e-9e52c6ae499c nodeName:}" failed. No retries permitted until 2026-09-16 16:10:09.133567045 +0000 UTC m=+1205.653189819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b" (UID: "c53b82f3-57d1-4af5-ad9e-9e52c6ae499c") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Sep 16 16:10:08.952226 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:08.952181    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:09.137209 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:10:09.137177    2574 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Sep 16 16:10:09.137382 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:10:09.137251    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tls-certs podName:c53b82f3-57d1-4af5-ad9e-9e52c6ae499c nodeName:}" failed. No retries permitted until 2026-09-16 16:10:10.137236018 +0000 UTC m=+1206.656858773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b" (UID: "c53b82f3-57d1-4af5-ad9e-9e52c6ae499c") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Sep 16 16:10:09.277455 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.277371    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:09.560475 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:10:09.560398    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:10:09.560475 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:10:09.560475 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:10:09.560475 ip-10-0-143-143 kubenswrapper[2574]:  > containerID="7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28" cmd=["/bin/sleep","15"]
Sep 16 16:10:09.560475 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.560443    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b" message="" err=<
Sep 16 16:10:09.560475 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:10:09.560475 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:10:09.560475 ip-10-0-143-143 kubenswrapper[2574]:  >
Sep 16 16:10:09.560475 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:10:09.560456    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:10:09.560475 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:10:09.560475 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:10:09.560475 ip-10-0-143-143 kubenswrapper[2574]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b" podUID="c53b82f3-57d1-4af5-ad9e-9e52c6ae499c" containerName="main" containerID="cri-o://7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28"
Sep 16 16:10:09.560974 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.560501    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b" podUID="c53b82f3-57d1-4af5-ad9e-9e52c6ae499c" containerName="main" containerID="cri-o://7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28" gracePeriod=60
Sep 16 16:10:09.841632 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.841565    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:09.944650 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.944619    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-dshm\") pod \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") "
Sep 16 16:10:09.944823 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.944705    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xw7fz\" (UniqueName: \"kubernetes.io/projected/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-kube-api-access-xw7fz\") pod \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") "
Sep 16 16:10:09.944823 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.944728    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tls-certs\") pod \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") "
Sep 16 16:10:09.944823 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.944755    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tmp-dir\") pod \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") "
Sep 16 16:10:09.944823 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.944815    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-home\") pod \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") "
Sep 16 16:10:09.945272 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.944838    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-model-cache\") pod \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\" (UID: \"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c\") "
Sep 16 16:10:09.945272 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.945040    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c53b82f3-57d1-4af5-ad9e-9e52c6ae499c" (UID: "c53b82f3-57d1-4af5-ad9e-9e52c6ae499c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:09.945272 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.945056    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-home" (OuterVolumeSpecName: "home") pod "c53b82f3-57d1-4af5-ad9e-9e52c6ae499c" (UID: "c53b82f3-57d1-4af5-ad9e-9e52c6ae499c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:09.945272 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.945111    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c53b82f3-57d1-4af5-ad9e-9e52c6ae499c" (UID: "c53b82f3-57d1-4af5-ad9e-9e52c6ae499c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:09.945272 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.945150    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:09.945272 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.945167    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:09.946995 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.946971    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-dshm" (OuterVolumeSpecName: "dshm") pod "c53b82f3-57d1-4af5-ad9e-9e52c6ae499c" (UID: "c53b82f3-57d1-4af5-ad9e-9e52c6ae499c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:09.946995 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.946989    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-kube-api-access-xw7fz" (OuterVolumeSpecName: "kube-api-access-xw7fz") pod "c53b82f3-57d1-4af5-ad9e-9e52c6ae499c" (UID: "c53b82f3-57d1-4af5-ad9e-9e52c6ae499c"). InnerVolumeSpecName "kube-api-access-xw7fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:10:09.947080 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.947018    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c53b82f3-57d1-4af5-ad9e-9e52c6ae499c" (UID: "c53b82f3-57d1-4af5-ad9e-9e52c6ae499c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:10:09.957072 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:09.957048    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:10.046507 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.046472    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:10.046507 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.046501    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:10.046507 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.046539    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw7fz\" (UniqueName: \"kubernetes.io/projected/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-kube-api-access-xw7fz\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:10.046764 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.046551    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:10.558108 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.558072    2574 generic.go:356] "Generic (PLEG): container finished" podID="c53b82f3-57d1-4af5-ad9e-9e52c6ae499c" containerID="7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28" exitCode=0
Sep 16 16:10:10.558291 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.558119    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"
Sep 16 16:10:10.558291 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.558156    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b" event={"ID":"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c","Type":"ContainerDied","Data":"7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28"}
Sep 16 16:10:10.558291 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.558205    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b" event={"ID":"c53b82f3-57d1-4af5-ad9e-9e52c6ae499c","Type":"ContainerDied","Data":"2fd2a40106e28c5674c21d5ccdd228edece4ba86bdabafe0410b0a55a958fc66"}
Sep 16 16:10:10.558291 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.558229    2574 scope.go:117] "RemoveContainer" containerID="7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28"
Sep 16 16:10:10.558589 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.558513    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:10.559677 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.559656    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:10.567145 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.566942    2574 scope.go:117] "RemoveContainer" containerID="7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28"
Sep 16 16:10:10.567225 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:10:10.567205    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28\": container with ID starting with 7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28 not found: ID does not exist" containerID="7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28"
Sep 16 16:10:10.567284 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.567230    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28"} err="failed to get container status \"7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28\": rpc error: code = NotFound desc = could not find container \"7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28\": container with ID starting with 7b36bce7ada782323e1ff5dd3ed1a0cc47c902ad15a73296bffd4aae2aa23b28 not found: ID does not exist"
Sep 16 16:10:10.573816 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.573790    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"]
Sep 16 16:10:10.577536 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:10.577503    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-np25b"]
Sep 16 16:10:11.996382 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:11.996348    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c53b82f3-57d1-4af5-ad9e-9e52c6ae499c" path="/var/lib/kubelet/pods/c53b82f3-57d1-4af5-ad9e-9e52c6ae499c/volumes"
Sep 16 16:10:12.288801 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:12.288717    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"]
Sep 16 16:10:12.288958 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:12.288918    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4" podUID="fdd75437-1890-4a13-88ec-5bb33d22a927" containerName="istio-proxy" containerID="cri-o://8db9ba04e37ee844aff750b7ed3ff2536d24f9c754bbba5188f9351afccaf76a" gracePeriod=30
Sep 16 16:10:14.439937 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.439914    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:10:14.582078 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.582048    2574 generic.go:356] "Generic (PLEG): container finished" podID="fdd75437-1890-4a13-88ec-5bb33d22a927" containerID="8db9ba04e37ee844aff750b7ed3ff2536d24f9c754bbba5188f9351afccaf76a" exitCode=0
Sep 16 16:10:14.582233 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.582093    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"
Sep 16 16:10:14.582233 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.582113    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4" event={"ID":"fdd75437-1890-4a13-88ec-5bb33d22a927","Type":"ContainerDied","Data":"8db9ba04e37ee844aff750b7ed3ff2536d24f9c754bbba5188f9351afccaf76a"}
Sep 16 16:10:14.582233 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.582153    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4" event={"ID":"fdd75437-1890-4a13-88ec-5bb33d22a927","Type":"ContainerDied","Data":"dbe0418af9fd5de9454b97ddc4f2dd6817c97d8bec77a36cfefa4fe53768b557"}
Sep 16 16:10:14.582233 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.582173    2574 scope.go:117] "RemoveContainer" containerID="8db9ba04e37ee844aff750b7ed3ff2536d24f9c754bbba5188f9351afccaf76a"
Sep 16 16:10:14.592509 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.592470    2574 scope.go:117] "RemoveContainer" containerID="8db9ba04e37ee844aff750b7ed3ff2536d24f9c754bbba5188f9351afccaf76a"
Sep 16 16:10:14.592997 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:10:14.592972    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8db9ba04e37ee844aff750b7ed3ff2536d24f9c754bbba5188f9351afccaf76a\": container with ID starting with 8db9ba04e37ee844aff750b7ed3ff2536d24f9c754bbba5188f9351afccaf76a not found: ID does not exist" containerID="8db9ba04e37ee844aff750b7ed3ff2536d24f9c754bbba5188f9351afccaf76a"
Sep 16 16:10:14.593086 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.593001    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8db9ba04e37ee844aff750b7ed3ff2536d24f9c754bbba5188f9351afccaf76a"} err="failed to get container status \"8db9ba04e37ee844aff750b7ed3ff2536d24f9c754bbba5188f9351afccaf76a\": rpc error: code = NotFound desc = could not find container \"8db9ba04e37ee844aff750b7ed3ff2536d24f9c754bbba5188f9351afccaf76a\": container with ID starting with 8db9ba04e37ee844aff750b7ed3ff2536d24f9c754bbba5188f9351afccaf76a not found: ID does not exist"
Sep 16 16:10:14.598290 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.598269    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-envoy\") pod \"fdd75437-1890-4a13-88ec-5bb33d22a927\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") "
Sep 16 16:10:14.598371 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.598300    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-data\") pod \"fdd75437-1890-4a13-88ec-5bb33d22a927\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") "
Sep 16 16:10:14.598371 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.598320    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-workload-certs\") pod \"fdd75437-1890-4a13-88ec-5bb33d22a927\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") "
Sep 16 16:10:14.598371 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.598347    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-credential-socket\") pod \"fdd75437-1890-4a13-88ec-5bb33d22a927\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") "
Sep 16 16:10:14.598540 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.598385    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-podinfo\") pod \"fdd75437-1890-4a13-88ec-5bb33d22a927\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") "
Sep 16 16:10:14.598540 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.598412    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd75437-1890-4a13-88ec-5bb33d22a927-istiod-ca-cert\") pod \"fdd75437-1890-4a13-88ec-5bb33d22a927\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") "
Sep 16 16:10:14.598540 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.598457    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-token\") pod \"fdd75437-1890-4a13-88ec-5bb33d22a927\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") "
Sep 16 16:10:14.598540 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.598482    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djqkh\" (UniqueName: \"kubernetes.io/projected/fdd75437-1890-4a13-88ec-5bb33d22a927-kube-api-access-djqkh\") pod \"fdd75437-1890-4a13-88ec-5bb33d22a927\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") "
Sep 16 16:10:14.598540 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.598538    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-workload-socket\") pod \"fdd75437-1890-4a13-88ec-5bb33d22a927\" (UID: \"fdd75437-1890-4a13-88ec-5bb33d22a927\") "
Sep 16 16:10:14.598797 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.598562    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fdd75437-1890-4a13-88ec-5bb33d22a927" (UID: "fdd75437-1890-4a13-88ec-5bb33d22a927"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:14.598797 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.598660    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-data" (OuterVolumeSpecName: "istio-data") pod "fdd75437-1890-4a13-88ec-5bb33d22a927" (UID: "fdd75437-1890-4a13-88ec-5bb33d22a927"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:14.598797 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.598754    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fdd75437-1890-4a13-88ec-5bb33d22a927" (UID: "fdd75437-1890-4a13-88ec-5bb33d22a927"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:14.598965 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.598838    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fdd75437-1890-4a13-88ec-5bb33d22a927" (UID: "fdd75437-1890-4a13-88ec-5bb33d22a927"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:14.598965 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.598848    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd75437-1890-4a13-88ec-5bb33d22a927-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fdd75437-1890-4a13-88ec-5bb33d22a927" (UID: "fdd75437-1890-4a13-88ec-5bb33d22a927"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:10:14.599074 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.599001    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-credential-socket\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:14.599074 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.599014    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd75437-1890-4a13-88ec-5bb33d22a927-istiod-ca-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:14.599074 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.599023    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-workload-socket\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:14.599074 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.599032    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-data\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:14.599074 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.599041    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-workload-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:14.600626 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.600597    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd75437-1890-4a13-88ec-5bb33d22a927-kube-api-access-djqkh" (OuterVolumeSpecName: "kube-api-access-djqkh") pod "fdd75437-1890-4a13-88ec-5bb33d22a927" (UID: "fdd75437-1890-4a13-88ec-5bb33d22a927"). InnerVolumeSpecName "kube-api-access-djqkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:10:14.600721 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.600690    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-token" (OuterVolumeSpecName: "istio-token") pod "fdd75437-1890-4a13-88ec-5bb33d22a927" (UID: "fdd75437-1890-4a13-88ec-5bb33d22a927"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:10:14.600908 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.600882    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fdd75437-1890-4a13-88ec-5bb33d22a927" (UID: "fdd75437-1890-4a13-88ec-5bb33d22a927"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:14.601016 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.600903    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fdd75437-1890-4a13-88ec-5bb33d22a927" (UID: "fdd75437-1890-4a13-88ec-5bb33d22a927"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 16:10:14.706652 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.700836    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-envoy\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:14.706652 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.700876    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-podinfo\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:14.706652 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.706636    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd75437-1890-4a13-88ec-5bb33d22a927-istio-token\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:14.706959 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.706673    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djqkh\" (UniqueName: \"kubernetes.io/projected/fdd75437-1890-4a13-88ec-5bb33d22a927-kube-api-access-djqkh\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:14.903342 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.903308    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"]
Sep 16 16:10:14.907300 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:14.907267    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-6q9l4"]
Sep 16 16:10:15.994388 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:15.994355    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdd75437-1890-4a13-88ec-5bb33d22a927" path="/var/lib/kubelet/pods/fdd75437-1890-4a13-88ec-5bb33d22a927/volumes"
Sep 16 16:10:35.079144 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.079109    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"]
Sep 16 16:10:35.079660 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.079529    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd75437-1890-4a13-88ec-5bb33d22a927" containerName="istio-proxy"
Sep 16 16:10:35.079660 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.079541    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd75437-1890-4a13-88ec-5bb33d22a927" containerName="istio-proxy"
Sep 16 16:10:35.079660 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.079548    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c53b82f3-57d1-4af5-ad9e-9e52c6ae499c" containerName="main"
Sep 16 16:10:35.079660 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.079553    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c53b82f3-57d1-4af5-ad9e-9e52c6ae499c" containerName="main"
Sep 16 16:10:35.079660 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.079633    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c53b82f3-57d1-4af5-ad9e-9e52c6ae499c" containerName="main"
Sep 16 16:10:35.079660 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.079645    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd75437-1890-4a13-88ec-5bb33d22a927" containerName="istio-proxy"
Sep 16 16:10:35.082711 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.082695    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.085775 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.085738    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-z55ld\""
Sep 16 16:10:35.085869 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.085803    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 16 16:10:35.091775 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.091750    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"]
Sep 16 16:10:35.193745 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.193713    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.193928 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.193768    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.193928 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.193794    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr7h7\" (UniqueName: \"kubernetes.io/projected/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-kube-api-access-jr7h7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.193928 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.193857    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.193928 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.193903    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.194068 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.193930    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.295148 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.295112    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.295148 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.295150    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.295356 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.295167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.295356 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.295190    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.295356 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.295227    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.295356 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.295253    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7h7\" (UniqueName: \"kubernetes.io/projected/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-kube-api-access-jr7h7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.295650 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.295623    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.295724 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.295629    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.295724 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.295689    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.297600 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.297575    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.297700 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.297680    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.302921 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.302899    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7h7\" (UniqueName: \"kubernetes.io/projected/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-kube-api-access-jr7h7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.394217 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.394181    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:35.674776 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.674639    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"]
Sep 16 16:10:35.689756 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:35.689718    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz" event={"ID":"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e","Type":"ContainerStarted","Data":"3ab84aa06be1a451de92ce893fbe0431b48836a3b7d030349ebc95658f6a362a"}
Sep 16 16:10:36.696352 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:36.696309    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz" event={"ID":"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e","Type":"ContainerStarted","Data":"35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056"}
Sep 16 16:10:36.715278 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:36.713324    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz" podStartSLOduration=1.7133048039999998 podStartE2EDuration="1.713304804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:10:36.711671172 +0000 UTC m=+1233.231293949" watchObservedRunningTime="2026-09-16 16:10:36.713304804 +0000 UTC m=+1233.232927583"
Sep 16 16:10:37.394400 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:37.394364    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:39.278821 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:39.278789    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"]
Sep 16 16:10:39.734545 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:10:39.734268    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:10:39.734545 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:10:39.734545 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:10:39.734545 ip-10-0-143-143 kubenswrapper[2574]:  > containerID="35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056" cmd=["/bin/sleep","15"]
Sep 16 16:10:39.734545 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:39.734331    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz" message="" err=<
Sep 16 16:10:39.734545 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:10:39.734545 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:10:39.734545 ip-10-0-143-143 kubenswrapper[2574]:  >
Sep 16 16:10:39.734545 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:10:39.734348    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:10:39.734545 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:10:39.734545 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:10:39.734545 ip-10-0-143-143 kubenswrapper[2574]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz" podUID="99c5d6e4-62d9-4c51-a505-f42c5f41ce4e" containerName="main" containerID="cri-o://35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056"
Sep 16 16:10:39.734545 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:39.734387    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz" podUID="99c5d6e4-62d9-4c51-a505-f42c5f41ce4e" containerName="main" containerID="cri-o://35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056" gracePeriod=60
Sep 16 16:10:40.093052 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.093030    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:40.245787 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.245737    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr7h7\" (UniqueName: \"kubernetes.io/projected/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-kube-api-access-jr7h7\") pod \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") "
Sep 16 16:10:40.245987 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.245799    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-tmp-dir\") pod \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") "
Sep 16 16:10:40.245987 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.245879    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-dshm\") pod \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") "
Sep 16 16:10:40.245987 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.245904    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-home\") pod \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") "
Sep 16 16:10:40.245987 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.245937    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-tls-certs\") pod \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") "
Sep 16 16:10:40.245987 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.245978    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-model-cache\") pod \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\" (UID: \"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e\") "
Sep 16 16:10:40.246230 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.246031    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99c5d6e4-62d9-4c51-a505-f42c5f41ce4e" (UID: "99c5d6e4-62d9-4c51-a505-f42c5f41ce4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:40.246230 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.246169    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-home" (OuterVolumeSpecName: "home") pod "99c5d6e4-62d9-4c51-a505-f42c5f41ce4e" (UID: "99c5d6e4-62d9-4c51-a505-f42c5f41ce4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:40.246230 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.246210    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "99c5d6e4-62d9-4c51-a505-f42c5f41ce4e" (UID: "99c5d6e4-62d9-4c51-a505-f42c5f41ce4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:40.246369 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.246347    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:40.246437 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.246369    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:40.246437 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.246384    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:40.248098 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.248071    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-dshm" (OuterVolumeSpecName: "dshm") pod "99c5d6e4-62d9-4c51-a505-f42c5f41ce4e" (UID: "99c5d6e4-62d9-4c51-a505-f42c5f41ce4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:40.248219 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.248118    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99c5d6e4-62d9-4c51-a505-f42c5f41ce4e" (UID: "99c5d6e4-62d9-4c51-a505-f42c5f41ce4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:10:40.248219 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.248187    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-kube-api-access-jr7h7" (OuterVolumeSpecName: "kube-api-access-jr7h7") pod "99c5d6e4-62d9-4c51-a505-f42c5f41ce4e" (UID: "99c5d6e4-62d9-4c51-a505-f42c5f41ce4e"). InnerVolumeSpecName "kube-api-access-jr7h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:10:40.347922 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.347893    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr7h7\" (UniqueName: \"kubernetes.io/projected/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-kube-api-access-jr7h7\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:40.347922 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.347920    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:40.347922 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.347929    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:40.720265 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.720228    2574 generic.go:356] "Generic (PLEG): container finished" podID="99c5d6e4-62d9-4c51-a505-f42c5f41ce4e" containerID="35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056" exitCode=0
Sep 16 16:10:40.720764 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.720362    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz" event={"ID":"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e","Type":"ContainerDied","Data":"35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056"}
Sep 16 16:10:40.720764 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.720587    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz" event={"ID":"99c5d6e4-62d9-4c51-a505-f42c5f41ce4e","Type":"ContainerDied","Data":"3ab84aa06be1a451de92ce893fbe0431b48836a3b7d030349ebc95658f6a362a"}
Sep 16 16:10:40.720764 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.720626    2574 scope.go:117] "RemoveContainer" containerID="35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056"
Sep 16 16:10:40.720764 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.720441    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"
Sep 16 16:10:40.735704 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.735672    2574 scope.go:117] "RemoveContainer" containerID="35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056"
Sep 16 16:10:40.736074 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:10:40.736048    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056\": container with ID starting with 35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056 not found: ID does not exist" containerID="35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056"
Sep 16 16:10:40.736185 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.736086    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056"} err="failed to get container status \"35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056\": rpc error: code = NotFound desc = could not find container \"35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056\": container with ID starting with 35af955bfddceb08e3995ec26c7c4b11f9e6084199b41ef9a14028889a1cf056 not found: ID does not exist"
Sep 16 16:10:40.753348 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.753302    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"]
Sep 16 16:10:40.757418 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:40.757388    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfd9fmz"]
Sep 16 16:10:41.994451 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:41.994418    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99c5d6e4-62d9-4c51-a505-f42c5f41ce4e" path="/var/lib/kubelet/pods/99c5d6e4-62d9-4c51-a505-f42c5f41ce4e/volumes"
Sep 16 16:10:42.305317 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:42.305221    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"]
Sep 16 16:10:42.305490 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:42.305467    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf" podUID="7e48a76b-6419-4df7-9484-7bf5271c61bc" containerName="istio-proxy" containerID="cri-o://e70e51830b9c0aad20eb372d3fe961b0bb4403a3b041fb8c44357f579505ea02" gracePeriod=30
Sep 16 16:10:44.742627 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.742601    2574 generic.go:356] "Generic (PLEG): container finished" podID="7e48a76b-6419-4df7-9484-7bf5271c61bc" containerID="e70e51830b9c0aad20eb372d3fe961b0bb4403a3b041fb8c44357f579505ea02" exitCode=0
Sep 16 16:10:44.742967 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.742669    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf" event={"ID":"7e48a76b-6419-4df7-9484-7bf5271c61bc","Type":"ContainerDied","Data":"e70e51830b9c0aad20eb372d3fe961b0bb4403a3b041fb8c44357f579505ea02"}
Sep 16 16:10:44.742967 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.742719    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf" event={"ID":"7e48a76b-6419-4df7-9484-7bf5271c61bc","Type":"ContainerDied","Data":"7abff80ce99c9c64566d8d68fb61cca6f05bfd522ae6a2791d4a2ff59c13105c"}
Sep 16 16:10:44.742967 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.742735    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7abff80ce99c9c64566d8d68fb61cca6f05bfd522ae6a2791d4a2ff59c13105c"
Sep 16 16:10:44.750472 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.750453    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:44.892583 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.892552    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-data\") pod \"7e48a76b-6419-4df7-9484-7bf5271c61bc\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") "
Sep 16 16:10:44.892583 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.892584    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-token\") pod \"7e48a76b-6419-4df7-9484-7bf5271c61bc\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") "
Sep 16 16:10:44.892787 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.892636    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-workload-certs\") pod \"7e48a76b-6419-4df7-9484-7bf5271c61bc\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") "
Sep 16 16:10:44.892787 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.892657    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-envoy\") pod \"7e48a76b-6419-4df7-9484-7bf5271c61bc\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") "
Sep 16 16:10:44.892888 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.892862    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmj98\" (UniqueName: \"kubernetes.io/projected/7e48a76b-6419-4df7-9484-7bf5271c61bc-kube-api-access-mmj98\") pod \"7e48a76b-6419-4df7-9484-7bf5271c61bc\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") "
Sep 16 16:10:44.892934 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.892840    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e48a76b-6419-4df7-9484-7bf5271c61bc" (UID: "7e48a76b-6419-4df7-9484-7bf5271c61bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:44.892934 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.892921    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e48a76b-6419-4df7-9484-7bf5271c61bc" (UID: "7e48a76b-6419-4df7-9484-7bf5271c61bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:44.893011 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.892940    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-credential-socket\") pod \"7e48a76b-6419-4df7-9484-7bf5271c61bc\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") "
Sep 16 16:10:44.893011 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.892980    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-podinfo\") pod \"7e48a76b-6419-4df7-9484-7bf5271c61bc\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") "
Sep 16 16:10:44.893120 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.893010    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e48a76b-6419-4df7-9484-7bf5271c61bc-istiod-ca-cert\") pod \"7e48a76b-6419-4df7-9484-7bf5271c61bc\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") "
Sep 16 16:10:44.893120 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.893055    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-workload-socket\") pod \"7e48a76b-6419-4df7-9484-7bf5271c61bc\" (UID: \"7e48a76b-6419-4df7-9484-7bf5271c61bc\") "
Sep 16 16:10:44.893210 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.893164    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e48a76b-6419-4df7-9484-7bf5271c61bc" (UID: "7e48a76b-6419-4df7-9484-7bf5271c61bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:44.893427 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.893391    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e48a76b-6419-4df7-9484-7bf5271c61bc" (UID: "7e48a76b-6419-4df7-9484-7bf5271c61bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:44.893427 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.893409    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e48a76b-6419-4df7-9484-7bf5271c61bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e48a76b-6419-4df7-9484-7bf5271c61bc" (UID: "7e48a76b-6419-4df7-9484-7bf5271c61bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:10:44.893427 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.893416    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-credential-socket\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:44.893721 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.893459    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-data\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:44.893721 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.893475    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-workload-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:44.894953 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.894922    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e48a76b-6419-4df7-9484-7bf5271c61bc" (UID: "7e48a76b-6419-4df7-9484-7bf5271c61bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:10:44.895047 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.894981    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e48a76b-6419-4df7-9484-7bf5271c61bc" (UID: "7e48a76b-6419-4df7-9484-7bf5271c61bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:10:44.895047 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.894995    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e48a76b-6419-4df7-9484-7bf5271c61bc-kube-api-access-mmj98" (OuterVolumeSpecName: "kube-api-access-mmj98") pod "7e48a76b-6419-4df7-9484-7bf5271c61bc" (UID: "7e48a76b-6419-4df7-9484-7bf5271c61bc"). InnerVolumeSpecName "kube-api-access-mmj98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:10:44.895047 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.894997    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e48a76b-6419-4df7-9484-7bf5271c61bc" (UID: "7e48a76b-6419-4df7-9484-7bf5271c61bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 16:10:44.994730 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.994698    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmj98\" (UniqueName: \"kubernetes.io/projected/7e48a76b-6419-4df7-9484-7bf5271c61bc-kube-api-access-mmj98\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:44.994730 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.994727    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-podinfo\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:44.994923 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.994738    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e48a76b-6419-4df7-9484-7bf5271c61bc-istiod-ca-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:44.994923 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.994751    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-workload-socket\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:44.994923 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.994764    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-token\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:44.994923 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:44.994775    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e48a76b-6419-4df7-9484-7bf5271c61bc-istio-envoy\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:10:45.754295 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:45.749513    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"
Sep 16 16:10:45.772714 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:45.772677    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"]
Sep 16 16:10:45.778912 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:45.778887    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-jcfqf"]
Sep 16 16:10:45.994384 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:45.994346    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e48a76b-6419-4df7-9484-7bf5271c61bc" path="/var/lib/kubelet/pods/7e48a76b-6419-4df7-9484-7bf5271c61bc/volumes"
Sep 16 16:10:52.488393 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.488361    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"]
Sep 16 16:10:52.488799 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.488742    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c5d6e4-62d9-4c51-a505-f42c5f41ce4e" containerName="main"
Sep 16 16:10:52.488799 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.488754    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c5d6e4-62d9-4c51-a505-f42c5f41ce4e" containerName="main"
Sep 16 16:10:52.488799 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.488768    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e48a76b-6419-4df7-9484-7bf5271c61bc" containerName="istio-proxy"
Sep 16 16:10:52.488799 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.488773    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e48a76b-6419-4df7-9484-7bf5271c61bc" containerName="istio-proxy"
Sep 16 16:10:52.488941 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.488836    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c5d6e4-62d9-4c51-a505-f42c5f41ce4e" containerName="main"
Sep 16 16:10:52.488941 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.488848    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e48a76b-6419-4df7-9484-7bf5271c61bc" containerName="istio-proxy"
Sep 16 16:10:52.493687 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.493667    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.496102 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.495970    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 16:10:52.496102 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.496089    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 16:10:52.496335 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.496132    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-kqxpg\""
Sep 16 16:10:52.496335 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.496188    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:10:52.496335 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.496272    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 16:10:52.496531 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.496348    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 16 16:10:52.504829 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.504808    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"]
Sep 16 16:10:52.668216 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.668175    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88sxr\" (UniqueName: \"kubernetes.io/projected/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-kube-api-access-88sxr\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.668407 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.668223    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-model-cache\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.668407 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.668256    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-tls-certs\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.668407 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.668316    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-tmp-dir\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.668407 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.668339    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-home\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.668407 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.668370    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-kserve-provision-location\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.668614 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.668448    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-dshm\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.668614 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.668486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-cabundle-cert\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.754692 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.754618    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"]
Sep 16 16:10:52.758440 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.758414    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"
Sep 16 16:10:52.760419 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.760396    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-xmbrh\""
Sep 16 16:10:52.767942 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.767912    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"]
Sep 16 16:10:52.771338 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.771309    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-kserve-provision-location\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.774937 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.774906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-dshm\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.775134 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.775112    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-cabundle-cert\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.776308 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.772421    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-kserve-provision-location\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.776308 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.775315    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88sxr\" (UniqueName: \"kubernetes.io/projected/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-kube-api-access-88sxr\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.776308 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.775413    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-model-cache\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.776308 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.775455    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-tls-certs\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.776308 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.775550    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-tmp-dir\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.776308 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.775594    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-home\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.776308 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.775927    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-home\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.776308 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.776117    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-cabundle-cert\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.776308 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.776202    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-model-cache\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.776308 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.776263    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-tmp-dir\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.778813 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.778791    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-tls-certs\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.779208 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.779185    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-dshm\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.783009 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.782988    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88sxr\" (UniqueName: \"kubernetes.io/projected/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-kube-api-access-88sxr\") pod \"auth-enabled-test-kserve-6fdc9775df-4dblm\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.805348 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.805317    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:10:52.877708 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.877441    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40c9122d-5adf-44c4-87ac-3b9eb21bd3ff-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr\" (UID: \"40c9122d-5adf-44c4-87ac-3b9eb21bd3ff\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"
Sep 16 16:10:52.877708 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.877586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktznx\" (UniqueName: \"kubernetes.io/projected/40c9122d-5adf-44c4-87ac-3b9eb21bd3ff-kube-api-access-ktznx\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr\" (UID: \"40c9122d-5adf-44c4-87ac-3b9eb21bd3ff\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"
Sep 16 16:10:52.978880 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.978838    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktznx\" (UniqueName: \"kubernetes.io/projected/40c9122d-5adf-44c4-87ac-3b9eb21bd3ff-kube-api-access-ktznx\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr\" (UID: \"40c9122d-5adf-44c4-87ac-3b9eb21bd3ff\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"
Sep 16 16:10:52.979047 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.978927    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40c9122d-5adf-44c4-87ac-3b9eb21bd3ff-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr\" (UID: \"40c9122d-5adf-44c4-87ac-3b9eb21bd3ff\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"
Sep 16 16:10:52.981295 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.981268    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40c9122d-5adf-44c4-87ac-3b9eb21bd3ff-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr\" (UID: \"40c9122d-5adf-44c4-87ac-3b9eb21bd3ff\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"
Sep 16 16:10:52.986611 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:52.986591    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktznx\" (UniqueName: \"kubernetes.io/projected/40c9122d-5adf-44c4-87ac-3b9eb21bd3ff-kube-api-access-ktznx\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr\" (UID: \"40c9122d-5adf-44c4-87ac-3b9eb21bd3ff\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"
Sep 16 16:10:53.075794 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:53.075701    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"
Sep 16 16:10:53.091800 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:53.091774    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"]
Sep 16 16:10:53.093074 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:10:53.093046    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0368296a_7022_4d48_aa0f_5adfa4a3b4d8.slice/crio-5706776db2e298cd38d39b190d426ccae4f29fba1249dea049c43a4f530acd49 WatchSource:0}: Error finding container 5706776db2e298cd38d39b190d426ccae4f29fba1249dea049c43a4f530acd49: Status 404 returned error can't find the container with id 5706776db2e298cd38d39b190d426ccae4f29fba1249dea049c43a4f530acd49
Sep 16 16:10:53.355716 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:53.355632    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"]
Sep 16 16:10:53.358227 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:10:53.358193    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c9122d_5adf_44c4_87ac_3b9eb21bd3ff.slice/crio-acfa6fdc2029378308afa16c3ef74e261d963637c0122135c0bad0ddf263220e WatchSource:0}: Error finding container acfa6fdc2029378308afa16c3ef74e261d963637c0122135c0bad0ddf263220e: Status 404 returned error can't find the container with id acfa6fdc2029378308afa16c3ef74e261d963637c0122135c0bad0ddf263220e
Sep 16 16:10:53.791942 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:53.791855    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr" event={"ID":"40c9122d-5adf-44c4-87ac-3b9eb21bd3ff","Type":"ContainerStarted","Data":"510cec1fb3824e51ffd2eb749185da96d68a364bb52fec724d3f6de7d3cbe566"}
Sep 16 16:10:53.791942 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:53.791913    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr" event={"ID":"40c9122d-5adf-44c4-87ac-3b9eb21bd3ff","Type":"ContainerStarted","Data":"acfa6fdc2029378308afa16c3ef74e261d963637c0122135c0bad0ddf263220e"}
Sep 16 16:10:53.792390 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:53.791953    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"
Sep 16 16:10:53.793583 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:53.793555    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" event={"ID":"0368296a-7022-4d48-aa0f-5adfa4a3b4d8","Type":"ContainerStarted","Data":"5179c19101a14055fc268741b33a8abc03d60d386f0f6b3e97ffc3645faba1e9"}
Sep 16 16:10:53.793700 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:53.793604    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" event={"ID":"0368296a-7022-4d48-aa0f-5adfa4a3b4d8","Type":"ContainerStarted","Data":"5706776db2e298cd38d39b190d426ccae4f29fba1249dea049c43a4f530acd49"}
Sep 16 16:10:53.808848 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:53.808801    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr" podStartSLOduration=1.808785289 podStartE2EDuration="1.808785289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:10:53.807149041 +0000 UTC m=+1250.326771817" watchObservedRunningTime="2026-09-16 16:10:53.808785289 +0000 UTC m=+1250.328408063"
Sep 16 16:10:55.804761 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:55.804665    2574 generic.go:356] "Generic (PLEG): container finished" podID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerID="5179c19101a14055fc268741b33a8abc03d60d386f0f6b3e97ffc3645faba1e9" exitCode=0
Sep 16 16:10:55.805101 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:55.804749    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" event={"ID":"0368296a-7022-4d48-aa0f-5adfa4a3b4d8","Type":"ContainerDied","Data":"5179c19101a14055fc268741b33a8abc03d60d386f0f6b3e97ffc3645faba1e9"}
Sep 16 16:10:56.811445 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:56.811411    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" event={"ID":"0368296a-7022-4d48-aa0f-5adfa4a3b4d8","Type":"ContainerStarted","Data":"ade48f7ec9af66a4fc2f36a2719ceac0f8be44aec6af1adb9796521e08dead0f"}
Sep 16 16:10:56.836178 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:56.836118    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" podStartSLOduration=2.609722097 podStartE2EDuration="4.836095545s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.226373448s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:10:56.833408144 +0000 UTC m=+1253.353030920" watchObservedRunningTime="2026-09-16 16:10:56.836095545 +0000 UTC m=+1253.355718321"
Sep 16 16:10:57.806351 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:10:57.806310    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:11:02.805589 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:11:02.805549    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:11:02.807179 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:11:02.807149    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 16 16:11:12.806459 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:11:12.806409    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 16 16:11:22.806720 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:11:22.806675    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 16 16:11:24.799992 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:11:24.799959    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"
Sep 16 16:11:32.805843 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:11:32.805761    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 16 16:11:42.806197 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:11:42.806150    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 16 16:11:52.806197 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:11:52.806148    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 16 16:12:02.806075 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:12:02.806035    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 16 16:12:12.806458 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:12:12.806415    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 16 16:12:22.806493 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:12:22.806448    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 16 16:12:32.806779 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:12:32.806683    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 16 16:12:42.806694 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:12:42.806648    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 16 16:12:52.816703 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:12:52.816667    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:12:52.824321 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:12:52.824298    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:12:57.710836 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:12:57.710802    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"]
Sep 16 16:12:57.719154 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:12:57.719125    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"]
Sep 16 16:13:04.799690 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:04.799648    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr" podUID="40c9122d-5adf-44c4-87ac-3b9eb21bd3ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:13:12.730194 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:12.730150    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr" podUID="40c9122d-5adf-44c4-87ac-3b9eb21bd3ff" containerName="main" containerID="cri-o://510cec1fb3824e51ffd2eb749185da96d68a364bb52fec724d3f6de7d3cbe566" gracePeriod=45
Sep 16 16:13:12.741306 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:12.741255    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main" containerID="cri-o://ade48f7ec9af66a4fc2f36a2719ceac0f8be44aec6af1adb9796521e08dead0f" gracePeriod=45
Sep 16 16:13:12.817571 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:12.817504    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Sep 16 16:13:13.407422 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.407397    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:13:13.518083 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.518000    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-model-cache\") pod \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") "
Sep 16 16:13:13.518083 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.518038    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-kserve-provision-location\") pod \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") "
Sep 16 16:13:13.518083 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.518068    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-dshm\") pod \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") "
Sep 16 16:13:13.518344 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.518096    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-cabundle-cert\") pod \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") "
Sep 16 16:13:13.518344 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.518119    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88sxr\" (UniqueName: \"kubernetes.io/projected/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-kube-api-access-88sxr\") pod \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") "
Sep 16 16:13:13.518344 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.518168    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-tls-certs\") pod \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") "
Sep 16 16:13:13.518344 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.518193    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-home\") pod \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") "
Sep 16 16:13:13.518344 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.518275    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-tmp-dir\") pod \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\" (UID: \"0368296a-7022-4d48-aa0f-5adfa4a3b4d8\") "
Sep 16 16:13:13.518344 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.518320    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "0368296a-7022-4d48-aa0f-5adfa4a3b4d8" (UID: "0368296a-7022-4d48-aa0f-5adfa4a3b4d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:13:13.518750 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.518575    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:13:13.518750 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.518642    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0368296a-7022-4d48-aa0f-5adfa4a3b4d8" (UID: "0368296a-7022-4d48-aa0f-5adfa4a3b4d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:13:13.519102 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.519071    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-home" (OuterVolumeSpecName: "home") pod "0368296a-7022-4d48-aa0f-5adfa4a3b4d8" (UID: "0368296a-7022-4d48-aa0f-5adfa4a3b4d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:13:13.520765 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.520740    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-dshm" (OuterVolumeSpecName: "dshm") pod "0368296a-7022-4d48-aa0f-5adfa4a3b4d8" (UID: "0368296a-7022-4d48-aa0f-5adfa4a3b4d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:13:13.520876 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.520758    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-kube-api-access-88sxr" (OuterVolumeSpecName: "kube-api-access-88sxr") pod "0368296a-7022-4d48-aa0f-5adfa4a3b4d8" (UID: "0368296a-7022-4d48-aa0f-5adfa4a3b4d8"). InnerVolumeSpecName "kube-api-access-88sxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:13:13.521165 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.521132    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0368296a-7022-4d48-aa0f-5adfa4a3b4d8" (UID: "0368296a-7022-4d48-aa0f-5adfa4a3b4d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:13:13.528483 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.528456    2574 generic.go:356] "Generic (PLEG): container finished" podID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerID="ade48f7ec9af66a4fc2f36a2719ceac0f8be44aec6af1adb9796521e08dead0f" exitCode=0
Sep 16 16:13:13.528616 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.528508    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" event={"ID":"0368296a-7022-4d48-aa0f-5adfa4a3b4d8","Type":"ContainerDied","Data":"ade48f7ec9af66a4fc2f36a2719ceac0f8be44aec6af1adb9796521e08dead0f"}
Sep 16 16:13:13.528616 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.528530    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"
Sep 16 16:13:13.528616 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.528578    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm" event={"ID":"0368296a-7022-4d48-aa0f-5adfa4a3b4d8","Type":"ContainerDied","Data":"5706776db2e298cd38d39b190d426ccae4f29fba1249dea049c43a4f530acd49"}
Sep 16 16:13:13.528616 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.528610    2574 scope.go:117] "RemoveContainer" containerID="ade48f7ec9af66a4fc2f36a2719ceac0f8be44aec6af1adb9796521e08dead0f"
Sep 16 16:13:13.532421 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.532399    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0368296a-7022-4d48-aa0f-5adfa4a3b4d8" (UID: "0368296a-7022-4d48-aa0f-5adfa4a3b4d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:13:13.539601 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.539581    2574 scope.go:117] "RemoveContainer" containerID="5179c19101a14055fc268741b33a8abc03d60d386f0f6b3e97ffc3645faba1e9"
Sep 16 16:13:13.542657 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.542632    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0368296a-7022-4d48-aa0f-5adfa4a3b4d8" (UID: "0368296a-7022-4d48-aa0f-5adfa4a3b4d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:13:13.547475 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.547458    2574 scope.go:117] "RemoveContainer" containerID="ade48f7ec9af66a4fc2f36a2719ceac0f8be44aec6af1adb9796521e08dead0f"
Sep 16 16:13:13.547771 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:13:13.547747    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade48f7ec9af66a4fc2f36a2719ceac0f8be44aec6af1adb9796521e08dead0f\": container with ID starting with ade48f7ec9af66a4fc2f36a2719ceac0f8be44aec6af1adb9796521e08dead0f not found: ID does not exist" containerID="ade48f7ec9af66a4fc2f36a2719ceac0f8be44aec6af1adb9796521e08dead0f"
Sep 16 16:13:13.547850 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.547777    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade48f7ec9af66a4fc2f36a2719ceac0f8be44aec6af1adb9796521e08dead0f"} err="failed to get container status \"ade48f7ec9af66a4fc2f36a2719ceac0f8be44aec6af1adb9796521e08dead0f\": rpc error: code = NotFound desc = could not find container \"ade48f7ec9af66a4fc2f36a2719ceac0f8be44aec6af1adb9796521e08dead0f\": container with ID starting with ade48f7ec9af66a4fc2f36a2719ceac0f8be44aec6af1adb9796521e08dead0f not found: ID does not exist"
Sep 16 16:13:13.547850 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.547793    2574 scope.go:117] "RemoveContainer" containerID="5179c19101a14055fc268741b33a8abc03d60d386f0f6b3e97ffc3645faba1e9"
Sep 16 16:13:13.548024 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:13:13.548009    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5179c19101a14055fc268741b33a8abc03d60d386f0f6b3e97ffc3645faba1e9\": container with ID starting with 5179c19101a14055fc268741b33a8abc03d60d386f0f6b3e97ffc3645faba1e9 not found: ID does not exist" containerID="5179c19101a14055fc268741b33a8abc03d60d386f0f6b3e97ffc3645faba1e9"
Sep 16 16:13:13.548064 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.548030    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5179c19101a14055fc268741b33a8abc03d60d386f0f6b3e97ffc3645faba1e9"} err="failed to get container status \"5179c19101a14055fc268741b33a8abc03d60d386f0f6b3e97ffc3645faba1e9\": rpc error: code = NotFound desc = could not find container \"5179c19101a14055fc268741b33a8abc03d60d386f0f6b3e97ffc3645faba1e9\": container with ID starting with 5179c19101a14055fc268741b33a8abc03d60d386f0f6b3e97ffc3645faba1e9 not found: ID does not exist"
Sep 16 16:13:13.620021 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.619991    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:13:13.620021 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.620016    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:13:13.620021 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.620025    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:13:13.620243 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.620035    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:13:13.620243 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.620045    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:13:13.620243 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.620054    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88sxr\" (UniqueName: \"kubernetes.io/projected/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-kube-api-access-88sxr\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:13:13.620243 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.620061    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0368296a-7022-4d48-aa0f-5adfa4a3b4d8-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:13:13.856594 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.856562    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"]
Sep 16 16:13:13.864949 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.864925    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fdc9775df-4dblm"]
Sep 16 16:13:13.994890 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:13.994862    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" path="/var/lib/kubelet/pods/0368296a-7022-4d48-aa0f-5adfa4a3b4d8/volumes"
Sep 16 16:13:14.799840 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:14.799804    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr" podUID="40c9122d-5adf-44c4-87ac-3b9eb21bd3ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:13:24.799903 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:24.799860    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr" podUID="40c9122d-5adf-44c4-87ac-3b9eb21bd3ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:13:24.800292 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:24.799999    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"
Sep 16 16:13:34.799737 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:34.799698    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr" podUID="40c9122d-5adf-44c4-87ac-3b9eb21bd3ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:13:42.970360 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:42.970331    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"
Sep 16 16:13:43.087345 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.087259    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40c9122d-5adf-44c4-87ac-3b9eb21bd3ff-tls-certs\") pod \"40c9122d-5adf-44c4-87ac-3b9eb21bd3ff\" (UID: \"40c9122d-5adf-44c4-87ac-3b9eb21bd3ff\") "
Sep 16 16:13:43.087345 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.087299    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktznx\" (UniqueName: \"kubernetes.io/projected/40c9122d-5adf-44c4-87ac-3b9eb21bd3ff-kube-api-access-ktznx\") pod \"40c9122d-5adf-44c4-87ac-3b9eb21bd3ff\" (UID: \"40c9122d-5adf-44c4-87ac-3b9eb21bd3ff\") "
Sep 16 16:13:43.089380 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.089347    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c9122d-5adf-44c4-87ac-3b9eb21bd3ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40c9122d-5adf-44c4-87ac-3b9eb21bd3ff" (UID: "40c9122d-5adf-44c4-87ac-3b9eb21bd3ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:13:43.089498 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.089482    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c9122d-5adf-44c4-87ac-3b9eb21bd3ff-kube-api-access-ktznx" (OuterVolumeSpecName: "kube-api-access-ktznx") pod "40c9122d-5adf-44c4-87ac-3b9eb21bd3ff" (UID: "40c9122d-5adf-44c4-87ac-3b9eb21bd3ff"). InnerVolumeSpecName "kube-api-access-ktznx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:13:43.188780 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.188741    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40c9122d-5adf-44c4-87ac-3b9eb21bd3ff-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:13:43.188780 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.188774    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktznx\" (UniqueName: \"kubernetes.io/projected/40c9122d-5adf-44c4-87ac-3b9eb21bd3ff-kube-api-access-ktznx\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:13:43.678837 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.678801    2574 generic.go:356] "Generic (PLEG): container finished" podID="40c9122d-5adf-44c4-87ac-3b9eb21bd3ff" containerID="510cec1fb3824e51ffd2eb749185da96d68a364bb52fec724d3f6de7d3cbe566" exitCode=0
Sep 16 16:13:43.679009 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.678843    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr" event={"ID":"40c9122d-5adf-44c4-87ac-3b9eb21bd3ff","Type":"ContainerDied","Data":"510cec1fb3824e51ffd2eb749185da96d68a364bb52fec724d3f6de7d3cbe566"}
Sep 16 16:13:43.679009 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.678887    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr" event={"ID":"40c9122d-5adf-44c4-87ac-3b9eb21bd3ff","Type":"ContainerDied","Data":"acfa6fdc2029378308afa16c3ef74e261d963637c0122135c0bad0ddf263220e"}
Sep 16 16:13:43.679009 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.678907    2574 scope.go:117] "RemoveContainer" containerID="510cec1fb3824e51ffd2eb749185da96d68a364bb52fec724d3f6de7d3cbe566"
Sep 16 16:13:43.679009 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.678847    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"
Sep 16 16:13:43.688055 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.688036    2574 scope.go:117] "RemoveContainer" containerID="510cec1fb3824e51ffd2eb749185da96d68a364bb52fec724d3f6de7d3cbe566"
Sep 16 16:13:43.688332 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:13:43.688312    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510cec1fb3824e51ffd2eb749185da96d68a364bb52fec724d3f6de7d3cbe566\": container with ID starting with 510cec1fb3824e51ffd2eb749185da96d68a364bb52fec724d3f6de7d3cbe566 not found: ID does not exist" containerID="510cec1fb3824e51ffd2eb749185da96d68a364bb52fec724d3f6de7d3cbe566"
Sep 16 16:13:43.688378 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.688339    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510cec1fb3824e51ffd2eb749185da96d68a364bb52fec724d3f6de7d3cbe566"} err="failed to get container status \"510cec1fb3824e51ffd2eb749185da96d68a364bb52fec724d3f6de7d3cbe566\": rpc error: code = NotFound desc = could not find container \"510cec1fb3824e51ffd2eb749185da96d68a364bb52fec724d3f6de7d3cbe566\": container with ID starting with 510cec1fb3824e51ffd2eb749185da96d68a364bb52fec724d3f6de7d3cbe566 not found: ID does not exist"
Sep 16 16:13:43.700173 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.700144    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"]
Sep 16 16:13:43.704475 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.704451    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-rjqcr"]
Sep 16 16:13:43.994850 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:43.994774    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40c9122d-5adf-44c4-87ac-3b9eb21bd3ff" path="/var/lib/kubelet/pods/40c9122d-5adf-44c4-87ac-3b9eb21bd3ff/volumes"
Sep 16 16:13:54.795265 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.795234    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"]
Sep 16 16:13:54.795677 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.795662    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c9122d-5adf-44c4-87ac-3b9eb21bd3ff" containerName="main"
Sep 16 16:13:54.795718 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.795680    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c9122d-5adf-44c4-87ac-3b9eb21bd3ff" containerName="main"
Sep 16 16:13:54.795718 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.795688    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main"
Sep 16 16:13:54.795718 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.795693    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main"
Sep 16 16:13:54.795718 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.795703    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="storage-initializer"
Sep 16 16:13:54.795718 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.795708    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="storage-initializer"
Sep 16 16:13:54.795864 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.795773    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0368296a-7022-4d48-aa0f-5adfa4a3b4d8" containerName="main"
Sep 16 16:13:54.795864 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.795782    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c9122d-5adf-44c4-87ac-3b9eb21bd3ff" containerName="main"
Sep 16 16:13:54.800395 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.800378    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.804027 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.804005    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:13:54.804317 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.804297    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 16 16:13:54.804471 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.804449    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 16:13:54.804621 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.804389    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 16:13:54.804621 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.804417    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 16:13:54.804621 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.804348    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-g2g48\""
Sep 16 16:13:54.812724 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.812704    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"]
Sep 16 16:13:54.889834 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.889806    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.889834 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.889836    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98mm8\" (UniqueName: \"kubernetes.io/projected/b6af7654-eb31-481e-8248-119c83ca4bc5-kube-api-access-98mm8\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.890007 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.889855    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6af7654-eb31-481e-8248-119c83ca4bc5-tls-certs\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.890007 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.889884    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-model-cache\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.890007 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.889949    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-home\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.890007 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.889974    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-dshm\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.890135 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.890088    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-tmp-dir\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.890135 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.890112    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6af7654-eb31-481e-8248-119c83ca4bc5-cabundle-cert\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.990632 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.990602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-model-cache\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.990801 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.990641    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-home\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.990801 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.990658    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-dshm\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.990801 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.990702    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-tmp-dir\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.990801 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.990718    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6af7654-eb31-481e-8248-119c83ca4bc5-cabundle-cert\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.990801 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.990742    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.990801 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.990761    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98mm8\" (UniqueName: \"kubernetes.io/projected/b6af7654-eb31-481e-8248-119c83ca4bc5-kube-api-access-98mm8\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.990801 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.990785    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6af7654-eb31-481e-8248-119c83ca4bc5-tls-certs\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.991056 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.990975    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-home\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.991292 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.991269    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-model-cache\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.991410 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.991292    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.991410 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.991396    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-tmp-dir\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.991503 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.991412    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6af7654-eb31-481e-8248-119c83ca4bc5-cabundle-cert\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.993121 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.993092    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-dshm\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.993338 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.993320    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6af7654-eb31-481e-8248-119c83ca4bc5-tls-certs\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:54.999062 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:54.999043    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98mm8\" (UniqueName: \"kubernetes.io/projected/b6af7654-eb31-481e-8248-119c83ca4bc5-kube-api-access-98mm8\") pod \"auth-invalid-token-test-kserve-69b946cdd5-wdvtf\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:55.110672 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:55.110640    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:13:55.386835 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:55.386805    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"]
Sep 16 16:13:55.388781 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:13:55.388746    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6af7654_eb31_481e_8248_119c83ca4bc5.slice/crio-3076b252053ab49adee10988805622ad058d6e16a68447b2933a1bde6e0ac252 WatchSource:0}: Error finding container 3076b252053ab49adee10988805622ad058d6e16a68447b2933a1bde6e0ac252: Status 404 returned error can't find the container with id 3076b252053ab49adee10988805622ad058d6e16a68447b2933a1bde6e0ac252
Sep 16 16:13:55.740136 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:55.740037    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" event={"ID":"b6af7654-eb31-481e-8248-119c83ca4bc5","Type":"ContainerStarted","Data":"f7d10edc1ef52ec69edbd600bef4c832606d92a20158b4b7e85fa433c8fa9281"}
Sep 16 16:13:55.740136 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:55.740102    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" event={"ID":"b6af7654-eb31-481e-8248-119c83ca4bc5","Type":"ContainerStarted","Data":"3076b252053ab49adee10988805622ad058d6e16a68447b2933a1bde6e0ac252"}
Sep 16 16:13:57.750581 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:57.750544    2574 generic.go:356] "Generic (PLEG): container finished" podID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerID="f7d10edc1ef52ec69edbd600bef4c832606d92a20158b4b7e85fa433c8fa9281" exitCode=0
Sep 16 16:13:57.750920 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:57.750555    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" event={"ID":"b6af7654-eb31-481e-8248-119c83ca4bc5","Type":"ContainerDied","Data":"f7d10edc1ef52ec69edbd600bef4c832606d92a20158b4b7e85fa433c8fa9281"}
Sep 16 16:13:58.757064 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:58.757023    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" event={"ID":"b6af7654-eb31-481e-8248-119c83ca4bc5","Type":"ContainerStarted","Data":"6c1fa145a51d48fcfab934e395f6cbfbe63350db3fb874cf20d5bc95fbce965e"}
Sep 16 16:13:58.781562 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:58.781488    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" podStartSLOduration=2.5790309689999997 podStartE2EDuration="4.781468592s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.202437623s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:13:58.779424121 +0000 UTC m=+1435.299046909" watchObservedRunningTime="2026-09-16 16:13:58.781468592 +0000 UTC m=+1435.301091367"
Sep 16 16:13:59.111396 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:13:59.111360    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:14:05.110877 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:14:05.110828    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:14:05.112634 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:14:05.112604    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 16:14:15.111779 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:14:15.111718    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 16:14:25.111913 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:14:25.111865    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 16:14:35.111226 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:14:35.111178    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 16:14:45.111256 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:14:45.111203    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 16:14:55.111258 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:14:55.111216    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 16:15:05.111358 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:05.111318    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 16:15:08.238733 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.238695    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"]
Sep 16 16:15:08.242295 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.242269    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"
Sep 16 16:15:08.245232 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.245200    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-tdtgz\""
Sep 16 16:15:08.245355 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.245200    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 16:15:08.246426 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.246402    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 16:15:08.246580 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.246431    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-9xtwm\""
Sep 16 16:15:08.246663 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.246611    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 16:15:08.255200 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.255177    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"]
Sep 16 16:15:08.306067 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.306034    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362aaff5-39fe-4e9e-ac49-e2c9cfc6833e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj\" (UID: \"362aaff5-39fe-4e9e-ac49-e2c9cfc6833e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"
Sep 16 16:15:08.306225 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.306091    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d54c2\" (UniqueName: \"kubernetes.io/projected/362aaff5-39fe-4e9e-ac49-e2c9cfc6833e-kube-api-access-d54c2\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj\" (UID: \"362aaff5-39fe-4e9e-ac49-e2c9cfc6833e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"
Sep 16 16:15:08.407034 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.406985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362aaff5-39fe-4e9e-ac49-e2c9cfc6833e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj\" (UID: \"362aaff5-39fe-4e9e-ac49-e2c9cfc6833e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"
Sep 16 16:15:08.407231 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.407051    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d54c2\" (UniqueName: \"kubernetes.io/projected/362aaff5-39fe-4e9e-ac49-e2c9cfc6833e-kube-api-access-d54c2\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj\" (UID: \"362aaff5-39fe-4e9e-ac49-e2c9cfc6833e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"
Sep 16 16:15:08.409784 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.409748    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362aaff5-39fe-4e9e-ac49-e2c9cfc6833e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj\" (UID: \"362aaff5-39fe-4e9e-ac49-e2c9cfc6833e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"
Sep 16 16:15:08.414847 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.414822    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d54c2\" (UniqueName: \"kubernetes.io/projected/362aaff5-39fe-4e9e-ac49-e2c9cfc6833e-kube-api-access-d54c2\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj\" (UID: \"362aaff5-39fe-4e9e-ac49-e2c9cfc6833e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"
Sep 16 16:15:08.553710 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.553615    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"
Sep 16 16:15:08.833187 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:08.833091    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"]
Sep 16 16:15:08.835620 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:15:08.835579    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362aaff5_39fe_4e9e_ac49_e2c9cfc6833e.slice/crio-5e85b63f434ef20730f23776026429f2d4c3808f93ba6de50d61b855d1fd6c6c WatchSource:0}: Error finding container 5e85b63f434ef20730f23776026429f2d4c3808f93ba6de50d61b855d1fd6c6c: Status 404 returned error can't find the container with id 5e85b63f434ef20730f23776026429f2d4c3808f93ba6de50d61b855d1fd6c6c
Sep 16 16:15:09.118822 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:09.118783    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj" event={"ID":"362aaff5-39fe-4e9e-ac49-e2c9cfc6833e","Type":"ContainerStarted","Data":"b2cab0d6eed176a521cafec6feb530d570e2272fa5b4d2ed7682677811ea5ed0"}
Sep 16 16:15:09.119016 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:09.118849    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj" event={"ID":"362aaff5-39fe-4e9e-ac49-e2c9cfc6833e","Type":"ContainerStarted","Data":"5e85b63f434ef20730f23776026429f2d4c3808f93ba6de50d61b855d1fd6c6c"}
Sep 16 16:15:09.119016 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:09.118898    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"
Sep 16 16:15:09.136575 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:09.136504    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj" podStartSLOduration=1.13648791 podStartE2EDuration="1.13648791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:15:09.135101685 +0000 UTC m=+1505.654724462" watchObservedRunningTime="2026-09-16 16:15:09.13648791 +0000 UTC m=+1505.656110686"
Sep 16 16:15:15.111808 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:15.111758    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 16:15:25.111496 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:25.111448    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 16:15:35.111737 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:35.111642    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 16:15:40.127262 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:40.127231    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"
Sep 16 16:15:41.460276 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:41.460241    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"]
Sep 16 16:15:45.111228 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:45.111184    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 16 16:15:50.126971 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:50.126933    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj" podUID="362aaff5-39fe-4e9e-ac49-e2c9cfc6833e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:15:53.172995 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.172961    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 16 16:15:53.173371 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:15:53.172997    2574 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj" podUID="362aaff5-39fe-4e9e-ac49-e2c9cfc6833e" containerName="main" containerID="cri-o://b2cab0d6eed176a521cafec6feb530d570e2272fa5b4d2ed7682677811ea5ed0"
Sep 16 16:15:53.173371 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.173048    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj" podUID="362aaff5-39fe-4e9e-ac49-e2c9cfc6833e" containerName="main" containerID="cri-o://b2cab0d6eed176a521cafec6feb530d570e2272fa5b4d2ed7682677811ea5ed0" gracePeriod=49
Sep 16 16:15:53.316233 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.316212    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj_362aaff5-39fe-4e9e-ac49-e2c9cfc6833e/main/0.log"
Sep 16 16:15:53.318402 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.318386    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"
Sep 16 16:15:53.360293 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.360265    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj_362aaff5-39fe-4e9e-ac49-e2c9cfc6833e/main/0.log"
Sep 16 16:15:53.360462 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.360308    2574 generic.go:356] "Generic (PLEG): container finished" podID="362aaff5-39fe-4e9e-ac49-e2c9cfc6833e" containerID="b2cab0d6eed176a521cafec6feb530d570e2272fa5b4d2ed7682677811ea5ed0" exitCode=1
Sep 16 16:15:53.360462 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.360347    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj" event={"ID":"362aaff5-39fe-4e9e-ac49-e2c9cfc6833e","Type":"ContainerDied","Data":"b2cab0d6eed176a521cafec6feb530d570e2272fa5b4d2ed7682677811ea5ed0"}
Sep 16 16:15:53.360462 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.360357    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"
Sep 16 16:15:53.360462 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.360397    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj" event={"ID":"362aaff5-39fe-4e9e-ac49-e2c9cfc6833e","Type":"ContainerDied","Data":"5e85b63f434ef20730f23776026429f2d4c3808f93ba6de50d61b855d1fd6c6c"}
Sep 16 16:15:53.360462 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.360426    2574 scope.go:117] "RemoveContainer" containerID="b2cab0d6eed176a521cafec6feb530d570e2272fa5b4d2ed7682677811ea5ed0"
Sep 16 16:15:53.370999 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.370980    2574 scope.go:117] "RemoveContainer" containerID="b2cab0d6eed176a521cafec6feb530d570e2272fa5b4d2ed7682677811ea5ed0"
Sep 16 16:15:53.371293 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:15:53.371275    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2cab0d6eed176a521cafec6feb530d570e2272fa5b4d2ed7682677811ea5ed0\": container with ID starting with b2cab0d6eed176a521cafec6feb530d570e2272fa5b4d2ed7682677811ea5ed0 not found: ID does not exist" containerID="b2cab0d6eed176a521cafec6feb530d570e2272fa5b4d2ed7682677811ea5ed0"
Sep 16 16:15:53.371340 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.371303    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2cab0d6eed176a521cafec6feb530d570e2272fa5b4d2ed7682677811ea5ed0"} err="failed to get container status \"b2cab0d6eed176a521cafec6feb530d570e2272fa5b4d2ed7682677811ea5ed0\": rpc error: code = NotFound desc = could not find container \"b2cab0d6eed176a521cafec6feb530d570e2272fa5b4d2ed7682677811ea5ed0\": container with ID starting with b2cab0d6eed176a521cafec6feb530d570e2272fa5b4d2ed7682677811ea5ed0 not found: ID does not exist"
Sep 16 16:15:53.424881 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.424797    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362aaff5-39fe-4e9e-ac49-e2c9cfc6833e-tls-certs\") pod \"362aaff5-39fe-4e9e-ac49-e2c9cfc6833e\" (UID: \"362aaff5-39fe-4e9e-ac49-e2c9cfc6833e\") "
Sep 16 16:15:53.425021 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.424942    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d54c2\" (UniqueName: \"kubernetes.io/projected/362aaff5-39fe-4e9e-ac49-e2c9cfc6833e-kube-api-access-d54c2\") pod \"362aaff5-39fe-4e9e-ac49-e2c9cfc6833e\" (UID: \"362aaff5-39fe-4e9e-ac49-e2c9cfc6833e\") "
Sep 16 16:15:53.426992 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.426968    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362aaff5-39fe-4e9e-ac49-e2c9cfc6833e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "362aaff5-39fe-4e9e-ac49-e2c9cfc6833e" (UID: "362aaff5-39fe-4e9e-ac49-e2c9cfc6833e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:15:53.427092 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.427035    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362aaff5-39fe-4e9e-ac49-e2c9cfc6833e-kube-api-access-d54c2" (OuterVolumeSpecName: "kube-api-access-d54c2") pod "362aaff5-39fe-4e9e-ac49-e2c9cfc6833e" (UID: "362aaff5-39fe-4e9e-ac49-e2c9cfc6833e"). InnerVolumeSpecName "kube-api-access-d54c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:15:53.525877 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.525847    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d54c2\" (UniqueName: \"kubernetes.io/projected/362aaff5-39fe-4e9e-ac49-e2c9cfc6833e-kube-api-access-d54c2\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:15:53.525877 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.525872    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362aaff5-39fe-4e9e-ac49-e2c9cfc6833e-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:15:53.683969 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.683895    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"]
Sep 16 16:15:53.689901 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.689875    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc54xdbj"]
Sep 16 16:15:53.995125 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:53.995047    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="362aaff5-39fe-4e9e-ac49-e2c9cfc6833e" path="/var/lib/kubelet/pods/362aaff5-39fe-4e9e-ac49-e2c9cfc6833e/volumes"
Sep 16 16:15:55.121145 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:55.121115    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:15:55.129241 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:55.129210    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:15:56.529136 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:15:56.529092    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"]
Sep 16 16:16:11.551215 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:11.551152    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main" containerID="cri-o://6c1fa145a51d48fcfab934e395f6cbfbe63350db3fb874cf20d5bc95fbce965e" gracePeriod=45
Sep 16 16:16:12.196266 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.196241    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:16:12.309225 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.309147    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98mm8\" (UniqueName: \"kubernetes.io/projected/b6af7654-eb31-481e-8248-119c83ca4bc5-kube-api-access-98mm8\") pod \"b6af7654-eb31-481e-8248-119c83ca4bc5\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") "
Sep 16 16:16:12.309225 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.309191    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-kserve-provision-location\") pod \"b6af7654-eb31-481e-8248-119c83ca4bc5\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") "
Sep 16 16:16:12.309225 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.309211    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6af7654-eb31-481e-8248-119c83ca4bc5-cabundle-cert\") pod \"b6af7654-eb31-481e-8248-119c83ca4bc5\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") "
Sep 16 16:16:12.309485 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.309236    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-tmp-dir\") pod \"b6af7654-eb31-481e-8248-119c83ca4bc5\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") "
Sep 16 16:16:12.309485 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.309268    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6af7654-eb31-481e-8248-119c83ca4bc5-tls-certs\") pod \"b6af7654-eb31-481e-8248-119c83ca4bc5\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") "
Sep 16 16:16:12.309485 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.309289    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-model-cache\") pod \"b6af7654-eb31-481e-8248-119c83ca4bc5\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") "
Sep 16 16:16:12.309485 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.309316    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-dshm\") pod \"b6af7654-eb31-481e-8248-119c83ca4bc5\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") "
Sep 16 16:16:12.309485 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.309377    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-home\") pod \"b6af7654-eb31-481e-8248-119c83ca4bc5\" (UID: \"b6af7654-eb31-481e-8248-119c83ca4bc5\") "
Sep 16 16:16:12.309757 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.309684    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6af7654-eb31-481e-8248-119c83ca4bc5" (UID: "b6af7654-eb31-481e-8248-119c83ca4bc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:16:12.309814 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.309739    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6af7654-eb31-481e-8248-119c83ca4bc5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b6af7654-eb31-481e-8248-119c83ca4bc5" (UID: "b6af7654-eb31-481e-8248-119c83ca4bc5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:16:12.310331 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.310303    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-home" (OuterVolumeSpecName: "home") pod "b6af7654-eb31-481e-8248-119c83ca4bc5" (UID: "b6af7654-eb31-481e-8248-119c83ca4bc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:16:12.311869 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.311821    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-dshm" (OuterVolumeSpecName: "dshm") pod "b6af7654-eb31-481e-8248-119c83ca4bc5" (UID: "b6af7654-eb31-481e-8248-119c83ca4bc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:16:12.311961 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.311929    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6af7654-eb31-481e-8248-119c83ca4bc5-kube-api-access-98mm8" (OuterVolumeSpecName: "kube-api-access-98mm8") pod "b6af7654-eb31-481e-8248-119c83ca4bc5" (UID: "b6af7654-eb31-481e-8248-119c83ca4bc5"). InnerVolumeSpecName "kube-api-access-98mm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:16:12.311961 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.311940    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6af7654-eb31-481e-8248-119c83ca4bc5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6af7654-eb31-481e-8248-119c83ca4bc5" (UID: "b6af7654-eb31-481e-8248-119c83ca4bc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:16:12.323061 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.323026    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6af7654-eb31-481e-8248-119c83ca4bc5" (UID: "b6af7654-eb31-481e-8248-119c83ca4bc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:16:12.332864 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.332839    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6af7654-eb31-481e-8248-119c83ca4bc5" (UID: "b6af7654-eb31-481e-8248-119c83ca4bc5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:16:12.410663 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.410628    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98mm8\" (UniqueName: \"kubernetes.io/projected/b6af7654-eb31-481e-8248-119c83ca4bc5-kube-api-access-98mm8\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:16:12.410663 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.410658    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:16:12.410864 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.410672    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6af7654-eb31-481e-8248-119c83ca4bc5-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:16:12.410864 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.410686    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:16:12.410864 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.410697    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6af7654-eb31-481e-8248-119c83ca4bc5-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:16:12.410864 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.410708    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:16:12.410864 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.410717    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:16:12.410864 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.410726    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6af7654-eb31-481e-8248-119c83ca4bc5-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:16:12.452258 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.452222    2574 generic.go:356] "Generic (PLEG): container finished" podID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerID="6c1fa145a51d48fcfab934e395f6cbfbe63350db3fb874cf20d5bc95fbce965e" exitCode=0
Sep 16 16:16:12.452258 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.452258    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" event={"ID":"b6af7654-eb31-481e-8248-119c83ca4bc5","Type":"ContainerDied","Data":"6c1fa145a51d48fcfab934e395f6cbfbe63350db3fb874cf20d5bc95fbce965e"}
Sep 16 16:16:12.452433 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.452270    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"
Sep 16 16:16:12.452433 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.452297    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" event={"ID":"b6af7654-eb31-481e-8248-119c83ca4bc5","Type":"ContainerDied","Data":"3076b252053ab49adee10988805622ad058d6e16a68447b2933a1bde6e0ac252"}
Sep 16 16:16:12.452433 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.452319    2574 scope.go:117] "RemoveContainer" containerID="6c1fa145a51d48fcfab934e395f6cbfbe63350db3fb874cf20d5bc95fbce965e"
Sep 16 16:16:12.461667 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.461651    2574 scope.go:117] "RemoveContainer" containerID="f7d10edc1ef52ec69edbd600bef4c832606d92a20158b4b7e85fa433c8fa9281"
Sep 16 16:16:12.469091 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.469071    2574 scope.go:117] "RemoveContainer" containerID="6c1fa145a51d48fcfab934e395f6cbfbe63350db3fb874cf20d5bc95fbce965e"
Sep 16 16:16:12.469345 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:16:12.469328    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c1fa145a51d48fcfab934e395f6cbfbe63350db3fb874cf20d5bc95fbce965e\": container with ID starting with 6c1fa145a51d48fcfab934e395f6cbfbe63350db3fb874cf20d5bc95fbce965e not found: ID does not exist" containerID="6c1fa145a51d48fcfab934e395f6cbfbe63350db3fb874cf20d5bc95fbce965e"
Sep 16 16:16:12.469391 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.469355    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c1fa145a51d48fcfab934e395f6cbfbe63350db3fb874cf20d5bc95fbce965e"} err="failed to get container status \"6c1fa145a51d48fcfab934e395f6cbfbe63350db3fb874cf20d5bc95fbce965e\": rpc error: code = NotFound desc = could not find container \"6c1fa145a51d48fcfab934e395f6cbfbe63350db3fb874cf20d5bc95fbce965e\": container with ID starting with 6c1fa145a51d48fcfab934e395f6cbfbe63350db3fb874cf20d5bc95fbce965e not found: ID does not exist"
Sep 16 16:16:12.469391 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.469372    2574 scope.go:117] "RemoveContainer" containerID="f7d10edc1ef52ec69edbd600bef4c832606d92a20158b4b7e85fa433c8fa9281"
Sep 16 16:16:12.469638 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:16:12.469608    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d10edc1ef52ec69edbd600bef4c832606d92a20158b4b7e85fa433c8fa9281\": container with ID starting with f7d10edc1ef52ec69edbd600bef4c832606d92a20158b4b7e85fa433c8fa9281 not found: ID does not exist" containerID="f7d10edc1ef52ec69edbd600bef4c832606d92a20158b4b7e85fa433c8fa9281"
Sep 16 16:16:12.469732 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.469636    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d10edc1ef52ec69edbd600bef4c832606d92a20158b4b7e85fa433c8fa9281"} err="failed to get container status \"f7d10edc1ef52ec69edbd600bef4c832606d92a20158b4b7e85fa433c8fa9281\": rpc error: code = NotFound desc = could not find container \"f7d10edc1ef52ec69edbd600bef4c832606d92a20158b4b7e85fa433c8fa9281\": container with ID starting with f7d10edc1ef52ec69edbd600bef4c832606d92a20158b4b7e85fa433c8fa9281 not found: ID does not exist"
Sep 16 16:16:12.474167 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.474145    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"]
Sep 16 16:16:12.478171 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:12.478151    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf"]
Sep 16 16:16:13.121943 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:13.121904    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-69b946cdd5-wdvtf" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 16:16:13.994340 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:16:13.994306    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" path="/var/lib/kubelet/pods/b6af7654-eb31-481e-8248-119c83ca4bc5/volumes"
Sep 16 16:17:04.134796 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:04.134759    2574 scope.go:117] "RemoveContainer" containerID="e70e51830b9c0aad20eb372d3fe961b0bb4403a3b041fb8c44357f579505ea02"
Sep 16 16:17:18.772314 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.772279    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"]
Sep 16 16:17:18.772901 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.772688    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362aaff5-39fe-4e9e-ac49-e2c9cfc6833e" containerName="main"
Sep 16 16:17:18.772901 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.772700    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="362aaff5-39fe-4e9e-ac49-e2c9cfc6833e" containerName="main"
Sep 16 16:17:18.772901 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.772710    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="storage-initializer"
Sep 16 16:17:18.772901 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.772716    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="storage-initializer"
Sep 16 16:17:18.772901 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.772739    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main"
Sep 16 16:17:18.772901 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.772744    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main"
Sep 16 16:17:18.772901 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.772806    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6af7654-eb31-481e-8248-119c83ca4bc5" containerName="main"
Sep 16 16:17:18.772901 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.772817    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="362aaff5-39fe-4e9e-ac49-e2c9cfc6833e" containerName="main"
Sep 16 16:17:18.776529 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.776497    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.779011 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.778986    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 16:17:18.779116 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.779028    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:17:18.779347 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.779307    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 16 16:17:18.779506 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.779359    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-f4lk7\""
Sep 16 16:17:18.779506 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.779373    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 16:17:18.779506 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.779418    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 16:17:18.790856 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.790833    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"]
Sep 16 16:17:18.813897 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.813862    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020cc236-a030-4c9f-a204-6e289efdd8e5-tls-certs\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.814098 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.813913    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m6rm\" (UniqueName: \"kubernetes.io/projected/020cc236-a030-4c9f-a204-6e289efdd8e5-kube-api-access-6m6rm\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.814098 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.813985    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-tmp-dir\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.814098 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.814042    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-dshm\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.814269 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.814138    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020cc236-a030-4c9f-a204-6e289efdd8e5-cabundle-cert\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.814269 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.814183    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-model-cache\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.814269 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.814213    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-home\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.814269 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.814239    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-kserve-provision-location\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.914804 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.914760    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-model-cache\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.914997 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.914948    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-home\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.914997 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.914983    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-kserve-provision-location\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.915134 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.915016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020cc236-a030-4c9f-a204-6e289efdd8e5-tls-certs\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.915134 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.915078    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m6rm\" (UniqueName: \"kubernetes.io/projected/020cc236-a030-4c9f-a204-6e289efdd8e5-kube-api-access-6m6rm\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.915134 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.915123    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-tmp-dir\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.915288 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.915158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-dshm\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.915288 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.915248    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-model-cache\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.915288 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.915252    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020cc236-a030-4c9f-a204-6e289efdd8e5-cabundle-cert\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.915440 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.915324    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-home\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.915440 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.915390    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-kserve-provision-location\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.915694 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.915676    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-tmp-dir\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.915898 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.915877    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020cc236-a030-4c9f-a204-6e289efdd8e5-cabundle-cert\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.917407 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.917378    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-dshm\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.917667 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.917649    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020cc236-a030-4c9f-a204-6e289efdd8e5-tls-certs\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:18.922466 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:18.922448    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m6rm\" (UniqueName: \"kubernetes.io/projected/020cc236-a030-4c9f-a204-6e289efdd8e5-kube-api-access-6m6rm\") pod \"auth-disabled-test-kserve-6bf644bb86-7cqrs\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:19.008887 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.008851    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"]
Sep 16 16:17:19.013726 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.013698    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"
Sep 16 16:17:19.016192 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.016061    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-nbp68\""
Sep 16 16:17:19.024667 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.024603    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"]
Sep 16 16:17:19.087123 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.087076    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:19.117466 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.117428    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dntq9\" (UniqueName: \"kubernetes.io/projected/42cef46b-df45-4ca9-98ef-2c239571decf-kube-api-access-dntq9\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc\" (UID: \"42cef46b-df45-4ca9-98ef-2c239571decf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"
Sep 16 16:17:19.117466 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.117471    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cef46b-df45-4ca9-98ef-2c239571decf-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc\" (UID: \"42cef46b-df45-4ca9-98ef-2c239571decf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"
Sep 16 16:17:19.218592 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.218550    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dntq9\" (UniqueName: \"kubernetes.io/projected/42cef46b-df45-4ca9-98ef-2c239571decf-kube-api-access-dntq9\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc\" (UID: \"42cef46b-df45-4ca9-98ef-2c239571decf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"
Sep 16 16:17:19.218787 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.218600    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cef46b-df45-4ca9-98ef-2c239571decf-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc\" (UID: \"42cef46b-df45-4ca9-98ef-2c239571decf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"
Sep 16 16:17:19.221095 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.221070    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cef46b-df45-4ca9-98ef-2c239571decf-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc\" (UID: \"42cef46b-df45-4ca9-98ef-2c239571decf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"
Sep 16 16:17:19.226547 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.226497    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dntq9\" (UniqueName: \"kubernetes.io/projected/42cef46b-df45-4ca9-98ef-2c239571decf-kube-api-access-dntq9\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc\" (UID: \"42cef46b-df45-4ca9-98ef-2c239571decf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"
Sep 16 16:17:19.325780 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.325634    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"
Sep 16 16:17:19.363287 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.363251    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"]
Sep 16 16:17:19.363921 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:17:19.363892    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020cc236_a030_4c9f_a204_6e289efdd8e5.slice/crio-437807de8e683cb5574d28d0533de4d457f29046413e0b855900a0d893e249ae WatchSource:0}: Error finding container 437807de8e683cb5574d28d0533de4d457f29046413e0b855900a0d893e249ae: Status 404 returned error can't find the container with id 437807de8e683cb5574d28d0533de4d457f29046413e0b855900a0d893e249ae
Sep 16 16:17:19.599792 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.599762    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"]
Sep 16 16:17:19.601971 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:17:19.601940    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42cef46b_df45_4ca9_98ef_2c239571decf.slice/crio-bbd283a18c2644795bb43f3ab3710a26740f7c851525ba8cb94cc48c70de25a5 WatchSource:0}: Error finding container bbd283a18c2644795bb43f3ab3710a26740f7c851525ba8cb94cc48c70de25a5: Status 404 returned error can't find the container with id bbd283a18c2644795bb43f3ab3710a26740f7c851525ba8cb94cc48c70de25a5
Sep 16 16:17:19.768741 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.768702    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" event={"ID":"020cc236-a030-4c9f-a204-6e289efdd8e5","Type":"ContainerStarted","Data":"d1aa20023c1e4d22058bbf7bbab221ee819bce9bbae1e5bde2a456105daaf5f0"}
Sep 16 16:17:19.768937 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.768767    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" event={"ID":"020cc236-a030-4c9f-a204-6e289efdd8e5","Type":"ContainerStarted","Data":"437807de8e683cb5574d28d0533de4d457f29046413e0b855900a0d893e249ae"}
Sep 16 16:17:19.770235 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.770208    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc" event={"ID":"42cef46b-df45-4ca9-98ef-2c239571decf","Type":"ContainerStarted","Data":"6956b4a1a19d3a0767fc19901ba57bdd0724897cede73a9f8ae8123d0c223cae"}
Sep 16 16:17:19.770354 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.770253    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc" event={"ID":"42cef46b-df45-4ca9-98ef-2c239571decf","Type":"ContainerStarted","Data":"bbd283a18c2644795bb43f3ab3710a26740f7c851525ba8cb94cc48c70de25a5"}
Sep 16 16:17:19.770354 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.770336    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"
Sep 16 16:17:19.809384 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:19.809321    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc" podStartSLOduration=1.809299241 podStartE2EDuration="1.809299241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:17:19.807467988 +0000 UTC m=+1636.327090763" watchObservedRunningTime="2026-09-16 16:17:19.809299241 +0000 UTC m=+1636.328922020"
Sep 16 16:17:22.786093 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:22.786057    2574 generic.go:356] "Generic (PLEG): container finished" podID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerID="d1aa20023c1e4d22058bbf7bbab221ee819bce9bbae1e5bde2a456105daaf5f0" exitCode=0
Sep 16 16:17:22.786483 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:22.786130    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" event={"ID":"020cc236-a030-4c9f-a204-6e289efdd8e5","Type":"ContainerDied","Data":"d1aa20023c1e4d22058bbf7bbab221ee819bce9bbae1e5bde2a456105daaf5f0"}
Sep 16 16:17:23.794104 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:23.794068    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" event={"ID":"020cc236-a030-4c9f-a204-6e289efdd8e5","Type":"ContainerStarted","Data":"e55a2dce680455914c5ee48cd9c07f7c0ec7ccfc9e757c2d4d8614d2baa16c48"}
Sep 16 16:17:23.816321 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:23.816264    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" podStartSLOduration=3.483119763 podStartE2EDuration="5.816248595s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.333128832s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:17:23.813625881 +0000 UTC m=+1640.333248657" watchObservedRunningTime="2026-09-16 16:17:23.816248595 +0000 UTC m=+1640.335871371"
Sep 16 16:17:24.088069 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:24.087987    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:29.088226 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:29.088191    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:17:29.089955 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:29.089928    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 16:17:39.088165 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:39.088103    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 16:17:49.087899 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:49.087851    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 16:17:50.776769 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:50.776740    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"
Sep 16 16:17:59.088452 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:17:59.088404    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 16:18:09.087797 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:18:09.087755    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 16:18:19.088026 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:18:19.087971    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 16:18:29.087666 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:18:29.087574    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 16:18:39.087812 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:18:39.087771    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 16:18:49.088233 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:18:49.088184    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 16:18:59.088270 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:18:59.088233    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 16:19:09.088015 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:09.087969    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 16:19:19.096824 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:19.096794    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:19:19.104156 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:19.104134    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:19:23.301782 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:23.301737    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"]
Sep 16 16:19:23.303906 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:23.303880    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"]
Sep 16 16:19:30.777134 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:30.777094    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc" podUID="42cef46b-df45-4ca9-98ef-2c239571decf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:19:38.321889 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:38.321838    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc" podUID="42cef46b-df45-4ca9-98ef-2c239571decf" containerName="main" containerID="cri-o://6956b4a1a19d3a0767fc19901ba57bdd0724897cede73a9f8ae8123d0c223cae" gracePeriod=45
Sep 16 16:19:38.326791 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:38.326752    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main" containerID="cri-o://e55a2dce680455914c5ee48cd9c07f7c0ec7ccfc9e757c2d4d8614d2baa16c48" gracePeriod=45
Sep 16 16:19:39.069357 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.069334    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:19:39.157070 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.157043    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-home\") pod \"020cc236-a030-4c9f-a204-6e289efdd8e5\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") "
Sep 16 16:19:39.157070 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.157074    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-tmp-dir\") pod \"020cc236-a030-4c9f-a204-6e289efdd8e5\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") "
Sep 16 16:19:39.157295 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.157093    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m6rm\" (UniqueName: \"kubernetes.io/projected/020cc236-a030-4c9f-a204-6e289efdd8e5-kube-api-access-6m6rm\") pod \"020cc236-a030-4c9f-a204-6e289efdd8e5\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") "
Sep 16 16:19:39.157295 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.157130    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-dshm\") pod \"020cc236-a030-4c9f-a204-6e289efdd8e5\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") "
Sep 16 16:19:39.157295 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.157163    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020cc236-a030-4c9f-a204-6e289efdd8e5-cabundle-cert\") pod \"020cc236-a030-4c9f-a204-6e289efdd8e5\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") "
Sep 16 16:19:39.157295 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.157185    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020cc236-a030-4c9f-a204-6e289efdd8e5-tls-certs\") pod \"020cc236-a030-4c9f-a204-6e289efdd8e5\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") "
Sep 16 16:19:39.157295 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.157219    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-kserve-provision-location\") pod \"020cc236-a030-4c9f-a204-6e289efdd8e5\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") "
Sep 16 16:19:39.157295 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.157255    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-model-cache\") pod \"020cc236-a030-4c9f-a204-6e289efdd8e5\" (UID: \"020cc236-a030-4c9f-a204-6e289efdd8e5\") "
Sep 16 16:19:39.157612 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.157512    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020cc236-a030-4c9f-a204-6e289efdd8e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "020cc236-a030-4c9f-a204-6e289efdd8e5" (UID: "020cc236-a030-4c9f-a204-6e289efdd8e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:19:39.157674 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.157661    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/020cc236-a030-4c9f-a204-6e289efdd8e5-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:19:39.157858 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.157818    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-home" (OuterVolumeSpecName: "home") pod "020cc236-a030-4c9f-a204-6e289efdd8e5" (UID: "020cc236-a030-4c9f-a204-6e289efdd8e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:19:39.157930 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.157888    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "020cc236-a030-4c9f-a204-6e289efdd8e5" (UID: "020cc236-a030-4c9f-a204-6e289efdd8e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:19:39.159416 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.159389    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-dshm" (OuterVolumeSpecName: "dshm") pod "020cc236-a030-4c9f-a204-6e289efdd8e5" (UID: "020cc236-a030-4c9f-a204-6e289efdd8e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:19:39.159745 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.159720    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/020cc236-a030-4c9f-a204-6e289efdd8e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "020cc236-a030-4c9f-a204-6e289efdd8e5" (UID: "020cc236-a030-4c9f-a204-6e289efdd8e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:19:39.159831 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.159723    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020cc236-a030-4c9f-a204-6e289efdd8e5-kube-api-access-6m6rm" (OuterVolumeSpecName: "kube-api-access-6m6rm") pod "020cc236-a030-4c9f-a204-6e289efdd8e5" (UID: "020cc236-a030-4c9f-a204-6e289efdd8e5"). InnerVolumeSpecName "kube-api-access-6m6rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:19:39.174425 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.174405    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "020cc236-a030-4c9f-a204-6e289efdd8e5" (UID: "020cc236-a030-4c9f-a204-6e289efdd8e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:19:39.181116 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.181095    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "020cc236-a030-4c9f-a204-6e289efdd8e5" (UID: "020cc236-a030-4c9f-a204-6e289efdd8e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:19:39.258266 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.258240    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020cc236-a030-4c9f-a204-6e289efdd8e5-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:19:39.258266 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.258262    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:19:39.258391 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.258272    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:19:39.258391 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.258281    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:19:39.258391 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.258288    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:19:39.258391 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.258295    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m6rm\" (UniqueName: \"kubernetes.io/projected/020cc236-a030-4c9f-a204-6e289efdd8e5-kube-api-access-6m6rm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:19:39.258391 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.258302    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020cc236-a030-4c9f-a204-6e289efdd8e5-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:19:39.486014 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.485929    2574 generic.go:356] "Generic (PLEG): container finished" podID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerID="e55a2dce680455914c5ee48cd9c07f7c0ec7ccfc9e757c2d4d8614d2baa16c48" exitCode=0
Sep 16 16:19:39.486014 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.485979    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" event={"ID":"020cc236-a030-4c9f-a204-6e289efdd8e5","Type":"ContainerDied","Data":"e55a2dce680455914c5ee48cd9c07f7c0ec7ccfc9e757c2d4d8614d2baa16c48"}
Sep 16 16:19:39.486014 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.485988    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"
Sep 16 16:19:39.486539 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.486050    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs" event={"ID":"020cc236-a030-4c9f-a204-6e289efdd8e5","Type":"ContainerDied","Data":"437807de8e683cb5574d28d0533de4d457f29046413e0b855900a0d893e249ae"}
Sep 16 16:19:39.486539 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.486073    2574 scope.go:117] "RemoveContainer" containerID="e55a2dce680455914c5ee48cd9c07f7c0ec7ccfc9e757c2d4d8614d2baa16c48"
Sep 16 16:19:39.495530 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.495502    2574 scope.go:117] "RemoveContainer" containerID="d1aa20023c1e4d22058bbf7bbab221ee819bce9bbae1e5bde2a456105daaf5f0"
Sep 16 16:19:39.502964 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.502950    2574 scope.go:117] "RemoveContainer" containerID="e55a2dce680455914c5ee48cd9c07f7c0ec7ccfc9e757c2d4d8614d2baa16c48"
Sep 16 16:19:39.503208 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:19:39.503191    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e55a2dce680455914c5ee48cd9c07f7c0ec7ccfc9e757c2d4d8614d2baa16c48\": container with ID starting with e55a2dce680455914c5ee48cd9c07f7c0ec7ccfc9e757c2d4d8614d2baa16c48 not found: ID does not exist" containerID="e55a2dce680455914c5ee48cd9c07f7c0ec7ccfc9e757c2d4d8614d2baa16c48"
Sep 16 16:19:39.503271 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.503214    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e55a2dce680455914c5ee48cd9c07f7c0ec7ccfc9e757c2d4d8614d2baa16c48"} err="failed to get container status \"e55a2dce680455914c5ee48cd9c07f7c0ec7ccfc9e757c2d4d8614d2baa16c48\": rpc error: code = NotFound desc = could not find container \"e55a2dce680455914c5ee48cd9c07f7c0ec7ccfc9e757c2d4d8614d2baa16c48\": container with ID starting with e55a2dce680455914c5ee48cd9c07f7c0ec7ccfc9e757c2d4d8614d2baa16c48 not found: ID does not exist"
Sep 16 16:19:39.503271 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.503230    2574 scope.go:117] "RemoveContainer" containerID="d1aa20023c1e4d22058bbf7bbab221ee819bce9bbae1e5bde2a456105daaf5f0"
Sep 16 16:19:39.503421 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:19:39.503406    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1aa20023c1e4d22058bbf7bbab221ee819bce9bbae1e5bde2a456105daaf5f0\": container with ID starting with d1aa20023c1e4d22058bbf7bbab221ee819bce9bbae1e5bde2a456105daaf5f0 not found: ID does not exist" containerID="d1aa20023c1e4d22058bbf7bbab221ee819bce9bbae1e5bde2a456105daaf5f0"
Sep 16 16:19:39.503459 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.503425    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1aa20023c1e4d22058bbf7bbab221ee819bce9bbae1e5bde2a456105daaf5f0"} err="failed to get container status \"d1aa20023c1e4d22058bbf7bbab221ee819bce9bbae1e5bde2a456105daaf5f0\": rpc error: code = NotFound desc = could not find container \"d1aa20023c1e4d22058bbf7bbab221ee819bce9bbae1e5bde2a456105daaf5f0\": container with ID starting with d1aa20023c1e4d22058bbf7bbab221ee819bce9bbae1e5bde2a456105daaf5f0 not found: ID does not exist"
Sep 16 16:19:39.513179 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.513157    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"]
Sep 16 16:19:39.517428 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.517407    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6bf644bb86-7cqrs"]
Sep 16 16:19:39.994239 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:39.994211    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" path="/var/lib/kubelet/pods/020cc236-a030-4c9f-a204-6e289efdd8e5/volumes"
Sep 16 16:19:40.776331 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:40.776298    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc" podUID="42cef46b-df45-4ca9-98ef-2c239571decf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:19:50.776632 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:50.776583    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc" podUID="42cef46b-df45-4ca9-98ef-2c239571decf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:19:50.777093 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:19:50.776706    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"
Sep 16 16:20:00.776734 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:00.776650    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc" podUID="42cef46b-df45-4ca9-98ef-2c239571decf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:20:08.559459 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.559434    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"
Sep 16 16:20:08.624008 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.623984    2574 generic.go:356] "Generic (PLEG): container finished" podID="42cef46b-df45-4ca9-98ef-2c239571decf" containerID="6956b4a1a19d3a0767fc19901ba57bdd0724897cede73a9f8ae8123d0c223cae" exitCode=0
Sep 16 16:20:08.624173 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.624024    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"
Sep 16 16:20:08.624173 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.624030    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc" event={"ID":"42cef46b-df45-4ca9-98ef-2c239571decf","Type":"ContainerDied","Data":"6956b4a1a19d3a0767fc19901ba57bdd0724897cede73a9f8ae8123d0c223cae"}
Sep 16 16:20:08.624173 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.624102    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc" event={"ID":"42cef46b-df45-4ca9-98ef-2c239571decf","Type":"ContainerDied","Data":"bbd283a18c2644795bb43f3ab3710a26740f7c851525ba8cb94cc48c70de25a5"}
Sep 16 16:20:08.624173 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.624133    2574 scope.go:117] "RemoveContainer" containerID="6956b4a1a19d3a0767fc19901ba57bdd0724897cede73a9f8ae8123d0c223cae"
Sep 16 16:20:08.632143 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.632127    2574 scope.go:117] "RemoveContainer" containerID="6956b4a1a19d3a0767fc19901ba57bdd0724897cede73a9f8ae8123d0c223cae"
Sep 16 16:20:08.632416 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:20:08.632392    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6956b4a1a19d3a0767fc19901ba57bdd0724897cede73a9f8ae8123d0c223cae\": container with ID starting with 6956b4a1a19d3a0767fc19901ba57bdd0724897cede73a9f8ae8123d0c223cae not found: ID does not exist" containerID="6956b4a1a19d3a0767fc19901ba57bdd0724897cede73a9f8ae8123d0c223cae"
Sep 16 16:20:08.632508 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.632419    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6956b4a1a19d3a0767fc19901ba57bdd0724897cede73a9f8ae8123d0c223cae"} err="failed to get container status \"6956b4a1a19d3a0767fc19901ba57bdd0724897cede73a9f8ae8123d0c223cae\": rpc error: code = NotFound desc = could not find container \"6956b4a1a19d3a0767fc19901ba57bdd0724897cede73a9f8ae8123d0c223cae\": container with ID starting with 6956b4a1a19d3a0767fc19901ba57bdd0724897cede73a9f8ae8123d0c223cae not found: ID does not exist"
Sep 16 16:20:08.701723 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.701701    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cef46b-df45-4ca9-98ef-2c239571decf-tls-certs\") pod \"42cef46b-df45-4ca9-98ef-2c239571decf\" (UID: \"42cef46b-df45-4ca9-98ef-2c239571decf\") "
Sep 16 16:20:08.701827 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.701786    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dntq9\" (UniqueName: \"kubernetes.io/projected/42cef46b-df45-4ca9-98ef-2c239571decf-kube-api-access-dntq9\") pod \"42cef46b-df45-4ca9-98ef-2c239571decf\" (UID: \"42cef46b-df45-4ca9-98ef-2c239571decf\") "
Sep 16 16:20:08.703648 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.703621    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42cef46b-df45-4ca9-98ef-2c239571decf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42cef46b-df45-4ca9-98ef-2c239571decf" (UID: "42cef46b-df45-4ca9-98ef-2c239571decf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:20:08.703743 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.703641    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42cef46b-df45-4ca9-98ef-2c239571decf-kube-api-access-dntq9" (OuterVolumeSpecName: "kube-api-access-dntq9") pod "42cef46b-df45-4ca9-98ef-2c239571decf" (UID: "42cef46b-df45-4ca9-98ef-2c239571decf"). InnerVolumeSpecName "kube-api-access-dntq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:20:08.802587 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.802565    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cef46b-df45-4ca9-98ef-2c239571decf-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:20:08.802587 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.802585    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dntq9\" (UniqueName: \"kubernetes.io/projected/42cef46b-df45-4ca9-98ef-2c239571decf-kube-api-access-dntq9\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:20:08.945309 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.945282    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"]
Sep 16 16:20:08.950144 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:08.950123    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-sfhvc"]
Sep 16 16:20:09.994474 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:09.994440    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42cef46b-df45-4ca9-98ef-2c239571decf" path="/var/lib/kubelet/pods/42cef46b-df45-4ca9-98ef-2c239571decf/volumes"
Sep 16 16:20:43.475921 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.475889    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"]
Sep 16 16:20:43.476328 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.476260    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="storage-initializer"
Sep 16 16:20:43.476328 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.476272    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="storage-initializer"
Sep 16 16:20:43.476328 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.476280    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42cef46b-df45-4ca9-98ef-2c239571decf" containerName="main"
Sep 16 16:20:43.476328 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.476285    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="42cef46b-df45-4ca9-98ef-2c239571decf" containerName="main"
Sep 16 16:20:43.476328 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.476302    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main"
Sep 16 16:20:43.476328 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.476307    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main"
Sep 16 16:20:43.476564 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.476380    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="020cc236-a030-4c9f-a204-6e289efdd8e5" containerName="main"
Sep 16 16:20:43.476564 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.476395    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="42cef46b-df45-4ca9-98ef-2c239571decf" containerName="main"
Sep 16 16:20:43.478979 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.478963    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.481089 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.481038    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 16 16:20:43.481895 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.481879    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 16 16:20:43.481994 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.481892    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 16 16:20:43.481994 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.481914    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-72nhm\""
Sep 16 16:20:43.488035 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.488012    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"]
Sep 16 16:20:43.495897 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.495878    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.496004 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.495903    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12746001-8529-45a9-967d-da68c1e6106e-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.496004 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.495922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqlqt\" (UniqueName: \"kubernetes.io/projected/12746001-8529-45a9-967d-da68c1e6106e-kube-api-access-qqlqt\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.496004 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.495939    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-dshm\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.496004 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.495960    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-model-cache\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.496162 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.496060    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-home\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.597490 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.597457    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.597490 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.597488    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12746001-8529-45a9-967d-da68c1e6106e-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.597685 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.597505    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqlqt\" (UniqueName: \"kubernetes.io/projected/12746001-8529-45a9-967d-da68c1e6106e-kube-api-access-qqlqt\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.597685 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.597539    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-dshm\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.597685 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.597574    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-model-cache\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.597685 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.597614    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-home\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.597930 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.597886    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.598002 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.597955    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-model-cache\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.598091 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.598059    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-home\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.599642 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.599622    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-dshm\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.599959 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.599941    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12746001-8529-45a9-967d-da68c1e6106e-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.605696 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.605672    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqlqt\" (UniqueName: \"kubernetes.io/projected/12746001-8529-45a9-967d-da68c1e6106e-kube-api-access-qqlqt\") pod \"canary-v1-kserve-65d58ccc9-6f9jh\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:43.789046 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:43.788982    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:44.064284 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:44.064257    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"]
Sep 16 16:20:44.066480 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:20:44.066454    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12746001_8529_45a9_967d_da68c1e6106e.slice/crio-770a488faa15b2ff0065f9d9690ceeacfb0c8e271c325ddfa5f62cfd50d2d27a WatchSource:0}: Error finding container 770a488faa15b2ff0065f9d9690ceeacfb0c8e271c325ddfa5f62cfd50d2d27a: Status 404 returned error can't find the container with id 770a488faa15b2ff0065f9d9690ceeacfb0c8e271c325ddfa5f62cfd50d2d27a
Sep 16 16:20:44.068399 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:44.068383    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 16:20:44.797827 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:44.797687    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" event={"ID":"12746001-8529-45a9-967d-da68c1e6106e","Type":"ContainerStarted","Data":"770a488faa15b2ff0065f9d9690ceeacfb0c8e271c325ddfa5f62cfd50d2d27a"}
Sep 16 16:20:45.803613 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:45.803576    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" event={"ID":"12746001-8529-45a9-967d-da68c1e6106e","Type":"ContainerStarted","Data":"eea2b4e4cea0ef104e65771a7bb194cad62fa7f221d75e80d9667d16d392abd6"}
Sep 16 16:20:45.823757 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:45.823710    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" podStartSLOduration=1.857050739 podStartE2EDuration="2.823694142s" totalImagesPullingTime="966.643403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:20:45.822213157 +0000 UTC m=+1842.341835934" watchObservedRunningTime="2026-09-16 16:20:45.823694142 +0000 UTC m=+1842.343316918"
Sep 16 16:20:46.789491 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:46.789457    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:53.789600 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:53.789562    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:53.801478 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:53.801449    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:20:53.853039 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:20:53.853013    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:21:46.816453 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:46.816420    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"]
Sep 16 16:21:46.820013 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:46.819993    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:46.822319 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:46.822286    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 16:21:46.823105 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:46.823085    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 16:21:46.823189 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:46.823085    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-6qvmv\""
Sep 16 16:21:46.823189 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:46.823132    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 16:21:46.831423 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:46.831399    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"]
Sep 16 16:21:46.939616 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:46.939578    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:46.939811 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:46.939641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:46.939811 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:46.939728    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:46.939903 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:46.939820    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:46.939903 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:46.939849    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfk66\" (UniqueName: \"kubernetes.io/projected/5342918f-326c-4c61-8333-d7ea14ac809c-kube-api-access-vfk66\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:46.939903 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:46.939887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5342918f-326c-4c61-8333-d7ea14ac809c-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:47.040751 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:47.040717    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:47.040932 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:47.040774    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:47.040932 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:47.040804    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:47.040932 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:47.040864    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:47.040932 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:47.040896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfk66\" (UniqueName: \"kubernetes.io/projected/5342918f-326c-4c61-8333-d7ea14ac809c-kube-api-access-vfk66\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:47.041147 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:47.041018    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5342918f-326c-4c61-8333-d7ea14ac809c-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:47.041201 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:47.041154    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:47.041201 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:47.041188    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:47.041297 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:47.041261    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:47.043021 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:47.042995    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:47.043330 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:47.043314    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5342918f-326c-4c61-8333-d7ea14ac809c-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:47.055294 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:47.055261    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfk66\" (UniqueName: \"kubernetes.io/projected/5342918f-326c-4c61-8333-d7ea14ac809c-kube-api-access-vfk66\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:47.130634 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:47.130597    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:47.403160 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:47.403134    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"]
Sep 16 16:21:47.405190 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:21:47.405162    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5342918f_326c_4c61_8333_d7ea14ac809c.slice/crio-117077dfc87c4651a080d85c7df23b23f3f8f368f3a5996887ee9aecdba380cb WatchSource:0}: Error finding container 117077dfc87c4651a080d85c7df23b23f3f8f368f3a5996887ee9aecdba380cb: Status 404 returned error can't find the container with id 117077dfc87c4651a080d85c7df23b23f3f8f368f3a5996887ee9aecdba380cb
Sep 16 16:21:48.092170 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:48.092126    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4" event={"ID":"5342918f-326c-4c61-8333-d7ea14ac809c","Type":"ContainerStarted","Data":"49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409"}
Sep 16 16:21:48.092664 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:48.092193    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4" event={"ID":"5342918f-326c-4c61-8333-d7ea14ac809c","Type":"ContainerStarted","Data":"117077dfc87c4651a080d85c7df23b23f3f8f368f3a5996887ee9aecdba380cb"}
Sep 16 16:21:48.109335 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:48.109274    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4" podStartSLOduration=2.109252342 podStartE2EDuration="2.109252342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:21:48.108328191 +0000 UTC m=+1904.627951004" watchObservedRunningTime="2026-09-16 16:21:48.109252342 +0000 UTC m=+1904.628875118"
Sep 16 16:21:48.131377 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:48.131329    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:57.131724 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:57.131688    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:57.144324 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:57.144300    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:21:57.155174 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:21:57.155153    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:22:20.687462 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:20.687426    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"]
Sep 16 16:22:20.696669 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:22:20.696631    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:22:20.696669 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:22:20.696669 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:22:20.696669 ip-10-0-143-143 kubenswrapper[2574]:  > containerID="49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409" cmd=["/bin/sleep","15"]
Sep 16 16:22:20.696918 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:20.696686    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4" message="" err=<
Sep 16 16:22:20.696918 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:22:20.696918 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:22:20.696918 ip-10-0-143-143 kubenswrapper[2574]:  >
Sep 16 16:22:20.696918 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:22:20.696698    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:22:20.696918 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:22:20.696918 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:22:20.696918 ip-10-0-143-143 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4" podUID="5342918f-326c-4c61-8333-d7ea14ac809c" containerName="main" containerID="cri-o://49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409"
Sep 16 16:22:20.696918 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:20.696733    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4" podUID="5342918f-326c-4c61-8333-d7ea14ac809c" containerName="main" containerID="cri-o://49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409" gracePeriod=60
Sep 16 16:22:21.048150 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.048124    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:22:21.149273 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.149234    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-home\") pod \"5342918f-326c-4c61-8333-d7ea14ac809c\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") "
Sep 16 16:22:21.149273 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.149276    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5342918f-326c-4c61-8333-d7ea14ac809c-tls-certs\") pod \"5342918f-326c-4c61-8333-d7ea14ac809c\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") "
Sep 16 16:22:21.149560 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.149321    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-model-cache\") pod \"5342918f-326c-4c61-8333-d7ea14ac809c\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") "
Sep 16 16:22:21.149560 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.149344    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-tmp-dir\") pod \"5342918f-326c-4c61-8333-d7ea14ac809c\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") "
Sep 16 16:22:21.149560 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.149373    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfk66\" (UniqueName: \"kubernetes.io/projected/5342918f-326c-4c61-8333-d7ea14ac809c-kube-api-access-vfk66\") pod \"5342918f-326c-4c61-8333-d7ea14ac809c\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") "
Sep 16 16:22:21.149560 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.149459    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-dshm\") pod \"5342918f-326c-4c61-8333-d7ea14ac809c\" (UID: \"5342918f-326c-4c61-8333-d7ea14ac809c\") "
Sep 16 16:22:21.149743 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.149577    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-home" (OuterVolumeSpecName: "home") pod "5342918f-326c-4c61-8333-d7ea14ac809c" (UID: "5342918f-326c-4c61-8333-d7ea14ac809c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:22:21.149743 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.149626    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5342918f-326c-4c61-8333-d7ea14ac809c" (UID: "5342918f-326c-4c61-8333-d7ea14ac809c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:22:21.149743 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.149694    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5342918f-326c-4c61-8333-d7ea14ac809c" (UID: "5342918f-326c-4c61-8333-d7ea14ac809c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:22:21.149882 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.149869    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:22:21.149934 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.149884    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:22:21.149934 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.149895    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:22:21.151695 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.151668    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5342918f-326c-4c61-8333-d7ea14ac809c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5342918f-326c-4c61-8333-d7ea14ac809c" (UID: "5342918f-326c-4c61-8333-d7ea14ac809c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:22:21.151695 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.151677    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-dshm" (OuterVolumeSpecName: "dshm") pod "5342918f-326c-4c61-8333-d7ea14ac809c" (UID: "5342918f-326c-4c61-8333-d7ea14ac809c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:22:21.151840 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.151691    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5342918f-326c-4c61-8333-d7ea14ac809c-kube-api-access-vfk66" (OuterVolumeSpecName: "kube-api-access-vfk66") pod "5342918f-326c-4c61-8333-d7ea14ac809c" (UID: "5342918f-326c-4c61-8333-d7ea14ac809c"). InnerVolumeSpecName "kube-api-access-vfk66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:22:21.250498 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.250412    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5342918f-326c-4c61-8333-d7ea14ac809c-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:22:21.250498 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.250445    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5342918f-326c-4c61-8333-d7ea14ac809c-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:22:21.250498 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.250455    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfk66\" (UniqueName: \"kubernetes.io/projected/5342918f-326c-4c61-8333-d7ea14ac809c-kube-api-access-vfk66\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:22:21.255919 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.255893    2574 generic.go:356] "Generic (PLEG): container finished" podID="5342918f-326c-4c61-8333-d7ea14ac809c" containerID="49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409" exitCode=0
Sep 16 16:22:21.256056 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.255939    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"
Sep 16 16:22:21.256056 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.255958    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4" event={"ID":"5342918f-326c-4c61-8333-d7ea14ac809c","Type":"ContainerDied","Data":"49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409"}
Sep 16 16:22:21.256056 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.255999    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4" event={"ID":"5342918f-326c-4c61-8333-d7ea14ac809c","Type":"ContainerDied","Data":"117077dfc87c4651a080d85c7df23b23f3f8f368f3a5996887ee9aecdba380cb"}
Sep 16 16:22:21.256056 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.256021    2574 scope.go:117] "RemoveContainer" containerID="49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409"
Sep 16 16:22:21.265179 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.265158    2574 scope.go:117] "RemoveContainer" containerID="49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409"
Sep 16 16:22:21.265428 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:22:21.265408    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409\": container with ID starting with 49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409 not found: ID does not exist" containerID="49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409"
Sep 16 16:22:21.265492 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.265438    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409"} err="failed to get container status \"49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409\": rpc error: code = NotFound desc = could not find container \"49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409\": container with ID starting with 49a5f6d9eb11a9595aa8b904c287df4fda47ea8bc174500d3d720251305c9409 not found: ID does not exist"
Sep 16 16:22:21.277139 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.277107    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"]
Sep 16 16:22:21.281405 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.281385    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444vdnn4"]
Sep 16 16:22:21.994206 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:22:21.994175    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5342918f-326c-4c61-8333-d7ea14ac809c" path="/var/lib/kubelet/pods/5342918f-326c-4c61-8333-d7ea14ac809c/volumes"
Sep 16 16:23:22.274548 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.274500    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"]
Sep 16 16:23:22.274926 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.274902    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5342918f-326c-4c61-8333-d7ea14ac809c" containerName="main"
Sep 16 16:23:22.274926 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.274914    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5342918f-326c-4c61-8333-d7ea14ac809c" containerName="main"
Sep 16 16:23:22.275015 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.275006    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5342918f-326c-4c61-8333-d7ea14ac809c" containerName="main"
Sep 16 16:23:22.278378 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.278359    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.281681 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.281656    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 16:23:22.281681 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.281675    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:23:22.281892 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.281722    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 16:23:22.281892 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.281722    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 16:23:22.281892 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.281652    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 16:23:22.281892 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.281656    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-qvcct\""
Sep 16 16:23:22.288153 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.288127    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"]
Sep 16 16:23:22.399215 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.399185    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.399215 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.399217    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-model-cache\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.399408 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.399237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-dshm\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.399408 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.399340    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rctmf\" (UniqueName: \"kubernetes.io/projected/3fa83640-172f-4b87-a70c-5ebc5e277e4b-kube-api-access-rctmf\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.399408 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.399372    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.399408 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.399402    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-home\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.399601 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.399463    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa83640-172f-4b87-a70c-5ebc5e277e4b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.399601 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.399496    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fa83640-172f-4b87-a70c-5ebc5e277e4b-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.500930 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.500899    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-dshm\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.501073 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.500947    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rctmf\" (UniqueName: \"kubernetes.io/projected/3fa83640-172f-4b87-a70c-5ebc5e277e4b-kube-api-access-rctmf\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.501073 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.500973    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.501073 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.501011    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-home\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.501073 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.501058    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa83640-172f-4b87-a70c-5ebc5e277e4b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.501284 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.501094    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fa83640-172f-4b87-a70c-5ebc5e277e4b-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.501284 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.501130    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.501284 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.501152    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-model-cache\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.501544 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.501458    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.501544 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.501487    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-home\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.501711 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.501585    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-model-cache\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.501770 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.501754    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.502003 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.501984    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fa83640-172f-4b87-a70c-5ebc5e277e4b-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.503886 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.503859    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-dshm\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.504272 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.504246    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa83640-172f-4b87-a70c-5ebc5e277e4b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.508622 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.508598    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rctmf\" (UniqueName: \"kubernetes.io/projected/3fa83640-172f-4b87-a70c-5ebc5e277e4b-kube-api-access-rctmf\") pod \"tokenizer-clean-path-test-kserve-84c665f65-fhrxz\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.514303 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.514274    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"]
Sep 16 16:23:22.518423 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.518405    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:23:22.533468 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.533440    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"]
Sep 16 16:23:22.589503 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.589473    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:22.601689 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.601665    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f88ce0a-3a84-499a-aae9-d7234479ae4f-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:23:22.601783 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.601709    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f88ce0a-3a84-499a-aae9-d7234479ae4f-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:23:22.601856 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.601832    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f88ce0a-3a84-499a-aae9-d7234479ae4f-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:23:22.601911 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.601895    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f88ce0a-3a84-499a-aae9-d7234479ae4f-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:23:22.703110 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.703079    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f88ce0a-3a84-499a-aae9-d7234479ae4f-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:23:22.703246 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.703157    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f88ce0a-3a84-499a-aae9-d7234479ae4f-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:23:22.703246 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.703197    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f88ce0a-3a84-499a-aae9-d7234479ae4f-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:23:22.703370 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.703262    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f88ce0a-3a84-499a-aae9-d7234479ae4f-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:23:22.703633 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.703610    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f88ce0a-3a84-499a-aae9-d7234479ae4f-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:23:22.703714 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.703665    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f88ce0a-3a84-499a-aae9-d7234479ae4f-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:23:22.703870 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.703853    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f88ce0a-3a84-499a-aae9-d7234479ae4f-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:23:22.705730 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.705711    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f88ce0a-3a84-499a-aae9-d7234479ae4f-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:23:22.829639 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.829577    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:23:22.862140 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:22.862117    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"]
Sep 16 16:23:22.864719 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:23:22.864696    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa83640_172f_4b87_a70c_5ebc5e277e4b.slice/crio-9e07d27d66c76b284fa9c28d5bc58bf63e4ad095c4a630a38c6e09a358752c3b WatchSource:0}: Error finding container 9e07d27d66c76b284fa9c28d5bc58bf63e4ad095c4a630a38c6e09a358752c3b: Status 404 returned error can't find the container with id 9e07d27d66c76b284fa9c28d5bc58bf63e4ad095c4a630a38c6e09a358752c3b
Sep 16 16:23:23.105645 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:23.105615    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"]
Sep 16 16:23:23.108093 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:23:23.108062    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f88ce0a_3a84_499a_aae9_d7234479ae4f.slice/crio-737c8e6ba8b76798f5b308ad3f7798bf62ad07a1d1fba9f93652d76e5c73f087 WatchSource:0}: Error finding container 737c8e6ba8b76798f5b308ad3f7798bf62ad07a1d1fba9f93652d76e5c73f087: Status 404 returned error can't find the container with id 737c8e6ba8b76798f5b308ad3f7798bf62ad07a1d1fba9f93652d76e5c73f087
Sep 16 16:23:23.560244 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:23.560158    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj" event={"ID":"8f88ce0a-3a84-499a-aae9-d7234479ae4f","Type":"ContainerStarted","Data":"9ead55c352e99476859377a7b0de71ad531a607b4a3ae480978f887e51a8b143"}
Sep 16 16:23:23.560244 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:23.560218    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj" event={"ID":"8f88ce0a-3a84-499a-aae9-d7234479ae4f","Type":"ContainerStarted","Data":"737c8e6ba8b76798f5b308ad3f7798bf62ad07a1d1fba9f93652d76e5c73f087"}
Sep 16 16:23:23.561889 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:23.561860    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz" event={"ID":"3fa83640-172f-4b87-a70c-5ebc5e277e4b","Type":"ContainerStarted","Data":"6418b0d2eedea946d0046a0e4d49fb65ce1875dddbe20281f06abc516fad5336"}
Sep 16 16:23:23.561982 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:23.561916    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz" event={"ID":"3fa83640-172f-4b87-a70c-5ebc5e277e4b","Type":"ContainerStarted","Data":"9e07d27d66c76b284fa9c28d5bc58bf63e4ad095c4a630a38c6e09a358752c3b"}
Sep 16 16:23:25.575021 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:25.574982    2574 generic.go:356] "Generic (PLEG): container finished" podID="3fa83640-172f-4b87-a70c-5ebc5e277e4b" containerID="6418b0d2eedea946d0046a0e4d49fb65ce1875dddbe20281f06abc516fad5336" exitCode=0
Sep 16 16:23:25.575535 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:25.575040    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz" event={"ID":"3fa83640-172f-4b87-a70c-5ebc5e277e4b","Type":"ContainerDied","Data":"6418b0d2eedea946d0046a0e4d49fb65ce1875dddbe20281f06abc516fad5336"}
Sep 16 16:23:26.580921 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:26.580887    2574 generic.go:356] "Generic (PLEG): container finished" podID="8f88ce0a-3a84-499a-aae9-d7234479ae4f" containerID="9ead55c352e99476859377a7b0de71ad531a607b4a3ae480978f887e51a8b143" exitCode=0
Sep 16 16:23:26.581373 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:26.580959    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj" event={"ID":"8f88ce0a-3a84-499a-aae9-d7234479ae4f","Type":"ContainerDied","Data":"9ead55c352e99476859377a7b0de71ad531a607b4a3ae480978f887e51a8b143"}
Sep 16 16:23:26.582845 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:26.582825    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz" event={"ID":"3fa83640-172f-4b87-a70c-5ebc5e277e4b","Type":"ContainerStarted","Data":"b37a66f4b9a21733176e41cded935689c54db2b47fcc5f00b7618fa967b91c9e"}
Sep 16 16:23:26.589937 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:26.589918    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:26.617898 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:26.617859    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz" podStartSLOduration=2.179494554 podStartE2EDuration="4.617843057s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.438348503s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:23:26.616654846 +0000 UTC m=+2003.136277634" watchObservedRunningTime="2026-09-16 16:23:26.617843057 +0000 UTC m=+2003.137465833"
Sep 16 16:23:32.589682 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:32.589639    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:32.604264 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:32.604234    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:32.636853 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:32.636773    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:23:51.958369 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:23:51.958337    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"]
Sep 16 16:24:06.979086 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:06.979001    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main" containerID="cri-o://eea2b4e4cea0ef104e65771a7bb194cad62fa7f221d75e80d9667d16d392abd6" gracePeriod=45
Sep 16 16:24:07.842840 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:07.842791    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 16 16:24:08.843132 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:08.843093    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 16 16:24:08.843540 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:08.843226    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:24:09.842593 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:09.842554    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 16 16:24:10.842834 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:10.842790    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 16 16:24:11.842949 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:11.842902    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 16 16:24:12.843158 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:12.843118    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 16 16:24:13.842819 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:13.842779    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 16 16:24:14.842816 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:14.842770    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 16 16:24:15.842924 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:15.842875    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 16 16:24:16.843291 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:16.843251    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Sep 16 16:24:17.895358 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.895324    2574 generic.go:356] "Generic (PLEG): container finished" podID="12746001-8529-45a9-967d-da68c1e6106e" containerID="eea2b4e4cea0ef104e65771a7bb194cad62fa7f221d75e80d9667d16d392abd6" exitCode=0
Sep 16 16:24:17.895718 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.895389    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" event={"ID":"12746001-8529-45a9-967d-da68c1e6106e","Type":"ContainerDied","Data":"eea2b4e4cea0ef104e65771a7bb194cad62fa7f221d75e80d9667d16d392abd6"}
Sep 16 16:24:17.895718 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.895441    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" event={"ID":"12746001-8529-45a9-967d-da68c1e6106e","Type":"ContainerDied","Data":"770a488faa15b2ff0065f9d9690ceeacfb0c8e271c325ddfa5f62cfd50d2d27a"}
Sep 16 16:24:17.895718 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.895457    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="770a488faa15b2ff0065f9d9690ceeacfb0c8e271c325ddfa5f62cfd50d2d27a"
Sep 16 16:24:17.918390 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.918364    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:24:17.989827 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.989790    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-tmp-dir\") pod \"12746001-8529-45a9-967d-da68c1e6106e\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") "
Sep 16 16:24:17.990011 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.989844    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-dshm\") pod \"12746001-8529-45a9-967d-da68c1e6106e\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") "
Sep 16 16:24:17.990011 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.989907    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-model-cache\") pod \"12746001-8529-45a9-967d-da68c1e6106e\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") "
Sep 16 16:24:17.990011 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.989984    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12746001-8529-45a9-967d-da68c1e6106e-tls-certs\") pod \"12746001-8529-45a9-967d-da68c1e6106e\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") "
Sep 16 16:24:17.990251 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.990035    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqlqt\" (UniqueName: \"kubernetes.io/projected/12746001-8529-45a9-967d-da68c1e6106e-kube-api-access-qqlqt\") pod \"12746001-8529-45a9-967d-da68c1e6106e\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") "
Sep 16 16:24:17.990251 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.990066    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12746001-8529-45a9-967d-da68c1e6106e" (UID: "12746001-8529-45a9-967d-da68c1e6106e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:17.990251 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.990087    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-home\") pod \"12746001-8529-45a9-967d-da68c1e6106e\" (UID: \"12746001-8529-45a9-967d-da68c1e6106e\") "
Sep 16 16:24:17.990251 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.990155    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-model-cache" (OuterVolumeSpecName: "model-cache") pod "12746001-8529-45a9-967d-da68c1e6106e" (UID: "12746001-8529-45a9-967d-da68c1e6106e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:17.990551 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.990328    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:17.990551 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.990348    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:17.990551 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.990367    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-home" (OuterVolumeSpecName: "home") pod "12746001-8529-45a9-967d-da68c1e6106e" (UID: "12746001-8529-45a9-967d-da68c1e6106e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:17.992296 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.992273    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-dshm" (OuterVolumeSpecName: "dshm") pod "12746001-8529-45a9-967d-da68c1e6106e" (UID: "12746001-8529-45a9-967d-da68c1e6106e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:17.992437 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.992418    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12746001-8529-45a9-967d-da68c1e6106e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12746001-8529-45a9-967d-da68c1e6106e" (UID: "12746001-8529-45a9-967d-da68c1e6106e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:24:17.992666 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:17.992637    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12746001-8529-45a9-967d-da68c1e6106e-kube-api-access-qqlqt" (OuterVolumeSpecName: "kube-api-access-qqlqt") pod "12746001-8529-45a9-967d-da68c1e6106e" (UID: "12746001-8529-45a9-967d-da68c1e6106e"). InnerVolumeSpecName "kube-api-access-qqlqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:24:18.091072 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:18.091041    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:18.091072 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:18.091067    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12746001-8529-45a9-967d-da68c1e6106e-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:18.091261 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:18.091079    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12746001-8529-45a9-967d-da68c1e6106e-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:18.091261 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:18.091091    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqlqt\" (UniqueName: \"kubernetes.io/projected/12746001-8529-45a9-967d-da68c1e6106e-kube-api-access-qqlqt\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:18.844068 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:18.844029    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 16:24:18.903164 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:18.903134    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"
Sep 16 16:24:18.903164 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:18.903156    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj" event={"ID":"8f88ce0a-3a84-499a-aae9-d7234479ae4f","Type":"ContainerStarted","Data":"647f0171a6550e8fd832be41d705f194eaad04e0f152edec70cd427a9fb886d7"}
Sep 16 16:24:18.903740 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:18.903481    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:24:18.923155 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:18.923109    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj" podStartSLOduration=3.009797439 podStartE2EDuration="56.923095101s" totalImagesPullingTime="51.450479264s" totalInitContainerRuntime="2.462818398s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:24:18.921245674 +0000 UTC m=+2055.440868449" watchObservedRunningTime="2026-09-16 16:24:18.923095101 +0000 UTC m=+2055.442717876"
Sep 16 16:24:18.933356 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:18.933328    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"]
Sep 16 16:24:18.938177 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:18.938150    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6f9jh"]
Sep 16 16:24:19.994396 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:19.994366    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12746001-8529-45a9-967d-da68c1e6106e" path="/var/lib/kubelet/pods/12746001-8529-45a9-967d-da68c1e6106e/volumes"
Sep 16 16:24:49.919611 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:49.919581    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:24:51.461562 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:51.461531    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj_8f88ce0a-3a84-499a-aae9-d7234479ae4f/main/0.log"
Sep 16 16:24:51.648714 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:51.648677    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"]
Sep 16 16:24:51.648905 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:51.648878    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj" podUID="8f88ce0a-3a84-499a-aae9-d7234479ae4f" containerName="main" containerID="cri-o://647f0171a6550e8fd832be41d705f194eaad04e0f152edec70cd427a9fb886d7" gracePeriod=30
Sep 16 16:24:51.651237 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:51.651205    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"]
Sep 16 16:24:51.659908 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:24:51.659879    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:24:51.659908 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:24:51.659908 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:24:51.659908 ip-10-0-143-143 kubenswrapper[2574]:  > containerID="b37a66f4b9a21733176e41cded935689c54db2b47fcc5f00b7618fa967b91c9e" cmd=["/bin/sleep","15"]
Sep 16 16:24:51.660180 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:51.659923    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz" message="" err=<
Sep 16 16:24:51.660180 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:24:51.660180 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:24:51.660180 ip-10-0-143-143 kubenswrapper[2574]:  >
Sep 16 16:24:51.660180 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:24:51.659935    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:24:51.660180 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:24:51.660180 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:24:51.660180 ip-10-0-143-143 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz" podUID="3fa83640-172f-4b87-a70c-5ebc5e277e4b" containerName="main" containerID="cri-o://b37a66f4b9a21733176e41cded935689c54db2b47fcc5f00b7618fa967b91c9e"
Sep 16 16:24:51.660180 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:51.659964    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz" podUID="3fa83640-172f-4b87-a70c-5ebc5e277e4b" containerName="main" containerID="cri-o://b37a66f4b9a21733176e41cded935689c54db2b47fcc5f00b7618fa967b91c9e" gracePeriod=60
Sep 16 16:24:52.077583 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.077476    2574 generic.go:356] "Generic (PLEG): container finished" podID="3fa83640-172f-4b87-a70c-5ebc5e277e4b" containerID="b37a66f4b9a21733176e41cded935689c54db2b47fcc5f00b7618fa967b91c9e" exitCode=0
Sep 16 16:24:52.077583 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.077556    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz" event={"ID":"3fa83640-172f-4b87-a70c-5ebc5e277e4b","Type":"ContainerDied","Data":"b37a66f4b9a21733176e41cded935689c54db2b47fcc5f00b7618fa967b91c9e"}
Sep 16 16:24:52.311742 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.311705    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:24:52.424938 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.424812    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f88ce0a-3a84-499a-aae9-d7234479ae4f-tls-certs\") pod \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") "
Sep 16 16:24:52.424938 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.424884    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f88ce0a-3a84-499a-aae9-d7234479ae4f-cabundle-cert\") pod \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") "
Sep 16 16:24:52.425091 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.424938    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f88ce0a-3a84-499a-aae9-d7234479ae4f-tmp-dir\") pod \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") "
Sep 16 16:24:52.425091 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.424975    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f88ce0a-3a84-499a-aae9-d7234479ae4f-kserve-provision-location\") pod \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\" (UID: \"8f88ce0a-3a84-499a-aae9-d7234479ae4f\") "
Sep 16 16:24:52.425562 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.425482    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f88ce0a-3a84-499a-aae9-d7234479ae4f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f88ce0a-3a84-499a-aae9-d7234479ae4f" (UID: "8f88ce0a-3a84-499a-aae9-d7234479ae4f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:24:52.426981 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.426953    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f88ce0a-3a84-499a-aae9-d7234479ae4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f88ce0a-3a84-499a-aae9-d7234479ae4f" (UID: "8f88ce0a-3a84-499a-aae9-d7234479ae4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:52.428108 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.428028    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f88ce0a-3a84-499a-aae9-d7234479ae4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f88ce0a-3a84-499a-aae9-d7234479ae4f" (UID: "8f88ce0a-3a84-499a-aae9-d7234479ae4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:24:52.437781 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.437754    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:24:52.463112 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.463074    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f88ce0a-3a84-499a-aae9-d7234479ae4f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f88ce0a-3a84-499a-aae9-d7234479ae4f" (UID: "8f88ce0a-3a84-499a-aae9-d7234479ae4f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:52.526367 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.526334    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rctmf\" (UniqueName: \"kubernetes.io/projected/3fa83640-172f-4b87-a70c-5ebc5e277e4b-kube-api-access-rctmf\") pod \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") "
Sep 16 16:24:52.526564 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.526378    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-model-cache\") pod \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") "
Sep 16 16:24:52.526564 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.526411    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-home\") pod \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") "
Sep 16 16:24:52.526564 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.526479    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-kserve-provision-location\") pod \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") "
Sep 16 16:24:52.526564 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.526510    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa83640-172f-4b87-a70c-5ebc5e277e4b-tls-certs\") pod \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") "
Sep 16 16:24:52.526564 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.526561    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-tmp-dir\") pod \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") "
Sep 16 16:24:52.526844 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.526680    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "3fa83640-172f-4b87-a70c-5ebc5e277e4b" (UID: "3fa83640-172f-4b87-a70c-5ebc5e277e4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:52.526844 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.526740    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fa83640-172f-4b87-a70c-5ebc5e277e4b-cabundle-cert\") pod \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") "
Sep 16 16:24:52.526844 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.526794    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3fa83640-172f-4b87-a70c-5ebc5e277e4b" (UID: "3fa83640-172f-4b87-a70c-5ebc5e277e4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:52.526844 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.526828    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-dshm\") pod \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\" (UID: \"3fa83640-172f-4b87-a70c-5ebc5e277e4b\") "
Sep 16 16:24:52.527068 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.527044    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa83640-172f-4b87-a70c-5ebc5e277e4b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3fa83640-172f-4b87-a70c-5ebc5e277e4b" (UID: "3fa83640-172f-4b87-a70c-5ebc5e277e4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:24:52.527133 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.527090    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-home" (OuterVolumeSpecName: "home") pod "3fa83640-172f-4b87-a70c-5ebc5e277e4b" (UID: "3fa83640-172f-4b87-a70c-5ebc5e277e4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:52.527190 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.527173    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.527240 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.527196    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.527240 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.527211    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f88ce0a-3a84-499a-aae9-d7234479ae4f-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.527240 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.527223    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f88ce0a-3a84-499a-aae9-d7234479ae4f-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.527240 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.527237    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f88ce0a-3a84-499a-aae9-d7234479ae4f-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.527445 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.527250    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.527445 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.527264    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fa83640-172f-4b87-a70c-5ebc5e277e4b-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.527445 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.527275    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f88ce0a-3a84-499a-aae9-d7234479ae4f-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.528731 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.528699    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa83640-172f-4b87-a70c-5ebc5e277e4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fa83640-172f-4b87-a70c-5ebc5e277e4b" (UID: "3fa83640-172f-4b87-a70c-5ebc5e277e4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:24:52.528731 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.528716    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa83640-172f-4b87-a70c-5ebc5e277e4b-kube-api-access-rctmf" (OuterVolumeSpecName: "kube-api-access-rctmf") pod "3fa83640-172f-4b87-a70c-5ebc5e277e4b" (UID: "3fa83640-172f-4b87-a70c-5ebc5e277e4b"). InnerVolumeSpecName "kube-api-access-rctmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:24:52.528893 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.528880    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-dshm" (OuterVolumeSpecName: "dshm") pod "3fa83640-172f-4b87-a70c-5ebc5e277e4b" (UID: "3fa83640-172f-4b87-a70c-5ebc5e277e4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:52.548006 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.547974    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3fa83640-172f-4b87-a70c-5ebc5e277e4b" (UID: "3fa83640-172f-4b87-a70c-5ebc5e277e4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:24:52.627751 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.627722    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.627751 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.627751    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa83640-172f-4b87-a70c-5ebc5e277e4b-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.627911 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.627763    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa83640-172f-4b87-a70c-5ebc5e277e4b-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:52.627911 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:52.627773    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rctmf\" (UniqueName: \"kubernetes.io/projected/3fa83640-172f-4b87-a70c-5ebc5e277e4b-kube-api-access-rctmf\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:24:53.084652 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.084568    2574 generic.go:356] "Generic (PLEG): container finished" podID="8f88ce0a-3a84-499a-aae9-d7234479ae4f" containerID="647f0171a6550e8fd832be41d705f194eaad04e0f152edec70cd427a9fb886d7" exitCode=0
Sep 16 16:24:53.084652 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.084609    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj" event={"ID":"8f88ce0a-3a84-499a-aae9-d7234479ae4f","Type":"ContainerDied","Data":"647f0171a6550e8fd832be41d705f194eaad04e0f152edec70cd427a9fb886d7"}
Sep 16 16:24:53.084652 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.084631    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"
Sep 16 16:24:53.084958 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.084671    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj" event={"ID":"8f88ce0a-3a84-499a-aae9-d7234479ae4f","Type":"ContainerDied","Data":"737c8e6ba8b76798f5b308ad3f7798bf62ad07a1d1fba9f93652d76e5c73f087"}
Sep 16 16:24:53.084958 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.084700    2574 scope.go:117] "RemoveContainer" containerID="647f0171a6550e8fd832be41d705f194eaad04e0f152edec70cd427a9fb886d7"
Sep 16 16:24:53.086595 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.086571    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz" event={"ID":"3fa83640-172f-4b87-a70c-5ebc5e277e4b","Type":"ContainerDied","Data":"9e07d27d66c76b284fa9c28d5bc58bf63e4ad095c4a630a38c6e09a358752c3b"}
Sep 16 16:24:53.086595 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.086590    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"
Sep 16 16:24:53.094679 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.094660    2574 scope.go:117] "RemoveContainer" containerID="9ead55c352e99476859377a7b0de71ad531a607b4a3ae480978f887e51a8b143"
Sep 16 16:24:53.102829 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.102814    2574 scope.go:117] "RemoveContainer" containerID="647f0171a6550e8fd832be41d705f194eaad04e0f152edec70cd427a9fb886d7"
Sep 16 16:24:53.103093 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:24:53.103074    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647f0171a6550e8fd832be41d705f194eaad04e0f152edec70cd427a9fb886d7\": container with ID starting with 647f0171a6550e8fd832be41d705f194eaad04e0f152edec70cd427a9fb886d7 not found: ID does not exist" containerID="647f0171a6550e8fd832be41d705f194eaad04e0f152edec70cd427a9fb886d7"
Sep 16 16:24:53.103166 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.103102    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647f0171a6550e8fd832be41d705f194eaad04e0f152edec70cd427a9fb886d7"} err="failed to get container status \"647f0171a6550e8fd832be41d705f194eaad04e0f152edec70cd427a9fb886d7\": rpc error: code = NotFound desc = could not find container \"647f0171a6550e8fd832be41d705f194eaad04e0f152edec70cd427a9fb886d7\": container with ID starting with 647f0171a6550e8fd832be41d705f194eaad04e0f152edec70cd427a9fb886d7 not found: ID does not exist"
Sep 16 16:24:53.103166 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.103120    2574 scope.go:117] "RemoveContainer" containerID="9ead55c352e99476859377a7b0de71ad531a607b4a3ae480978f887e51a8b143"
Sep 16 16:24:53.103376 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:24:53.103358    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ead55c352e99476859377a7b0de71ad531a607b4a3ae480978f887e51a8b143\": container with ID starting with 9ead55c352e99476859377a7b0de71ad531a607b4a3ae480978f887e51a8b143 not found: ID does not exist" containerID="9ead55c352e99476859377a7b0de71ad531a607b4a3ae480978f887e51a8b143"
Sep 16 16:24:53.103451 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.103380    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ead55c352e99476859377a7b0de71ad531a607b4a3ae480978f887e51a8b143"} err="failed to get container status \"9ead55c352e99476859377a7b0de71ad531a607b4a3ae480978f887e51a8b143\": rpc error: code = NotFound desc = could not find container \"9ead55c352e99476859377a7b0de71ad531a607b4a3ae480978f887e51a8b143\": container with ID starting with 9ead55c352e99476859377a7b0de71ad531a607b4a3ae480978f887e51a8b143 not found: ID does not exist"
Sep 16 16:24:53.103451 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.103395    2574 scope.go:117] "RemoveContainer" containerID="b37a66f4b9a21733176e41cded935689c54db2b47fcc5f00b7618fa967b91c9e"
Sep 16 16:24:53.111904 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.111884    2574 scope.go:117] "RemoveContainer" containerID="6418b0d2eedea946d0046a0e4d49fb65ce1875dddbe20281f06abc516fad5336"
Sep 16 16:24:53.119209 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.119183    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"]
Sep 16 16:24:53.124966 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.124943    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-84c665f65-fhrxz"]
Sep 16 16:24:53.137827 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.137799    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"]
Sep 16 16:24:53.146408 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.146385    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7558ff9b5b-jnvxj"]
Sep 16 16:24:53.994016 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.993988    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fa83640-172f-4b87-a70c-5ebc5e277e4b" path="/var/lib/kubelet/pods/3fa83640-172f-4b87-a70c-5ebc5e277e4b/volumes"
Sep 16 16:24:53.994424 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:53.994405    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f88ce0a-3a84-499a-aae9-d7234479ae4f" path="/var/lib/kubelet/pods/8f88ce0a-3a84-499a-aae9-d7234479ae4f/volumes"
Sep 16 16:24:57.968958 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.968916    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"]
Sep 16 16:24:57.969420 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.969395    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f88ce0a-3a84-499a-aae9-d7234479ae4f" containerName="storage-initializer"
Sep 16 16:24:57.969420 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.969420    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f88ce0a-3a84-499a-aae9-d7234479ae4f" containerName="storage-initializer"
Sep 16 16:24:57.969648 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.969463    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa83640-172f-4b87-a70c-5ebc5e277e4b" containerName="main"
Sep 16 16:24:57.969648 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.969472    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa83640-172f-4b87-a70c-5ebc5e277e4b" containerName="main"
Sep 16 16:24:57.969648 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.969480    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f88ce0a-3a84-499a-aae9-d7234479ae4f" containerName="main"
Sep 16 16:24:57.969648 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.969488    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f88ce0a-3a84-499a-aae9-d7234479ae4f" containerName="main"
Sep 16 16:24:57.969648 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.969504    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa83640-172f-4b87-a70c-5ebc5e277e4b" containerName="storage-initializer"
Sep 16 16:24:57.969648 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.969512    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa83640-172f-4b87-a70c-5ebc5e277e4b" containerName="storage-initializer"
Sep 16 16:24:57.969648 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.969538    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main"
Sep 16 16:24:57.969648 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.969547    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main"
Sep 16 16:24:57.969648 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.969635    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa83640-172f-4b87-a70c-5ebc5e277e4b" containerName="main"
Sep 16 16:24:57.969648 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.969648    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="12746001-8529-45a9-967d-da68c1e6106e" containerName="main"
Sep 16 16:24:57.970109 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.969661    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f88ce0a-3a84-499a-aae9-d7234479ae4f" containerName="main"
Sep 16 16:24:57.974651 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.974629    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:57.976987 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.976966    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 16 16:24:57.977740 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.977722    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 16 16:24:57.977840 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.977723    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-fbknw\""
Sep 16 16:24:57.977840 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.977807    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 16:24:57.982777 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:57.982746    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"]
Sep 16 16:24:58.079492 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.079456    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-model-cache\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.079492 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.079494    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-dshm\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.079734 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.079550    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16dcb650-2717-47f9-bbd4-941bb2b6ba18-tls-certs\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.079734 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.079594    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fbr\" (UniqueName: \"kubernetes.io/projected/16dcb650-2717-47f9-bbd4-941bb2b6ba18-kube-api-access-m5fbr\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.079734 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.079646    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.079734 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.079671    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-home\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.180878 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.180846    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-home\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.181056 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.180899    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-model-cache\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.181056 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.181023    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-dshm\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.181149 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.181071    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16dcb650-2717-47f9-bbd4-941bb2b6ba18-tls-certs\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.181149 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.181124    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fbr\" (UniqueName: \"kubernetes.io/projected/16dcb650-2717-47f9-bbd4-941bb2b6ba18-kube-api-access-m5fbr\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.181266 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.181160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.181318 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.181282    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-model-cache\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.181415 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.181391    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-home\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.181478 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.181458    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.183167 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.183139    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-dshm\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.183462 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.183446    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16dcb650-2717-47f9-bbd4-941bb2b6ba18-tls-certs\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.188607 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.188590    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fbr\" (UniqueName: \"kubernetes.io/projected/16dcb650-2717-47f9-bbd4-941bb2b6ba18-kube-api-access-m5fbr\") pod \"test-div-v1-kserve-fb566d667-kphtk\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.285669 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.285578    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:24:58.557561 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:58.557535    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"]
Sep 16 16:24:58.560069 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:24:58.560027    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16dcb650_2717_47f9_bbd4_941bb2b6ba18.slice/crio-25cae02710136b6bf774260d9e76346160dbc05b156fb6864c58493b757d3c60 WatchSource:0}: Error finding container 25cae02710136b6bf774260d9e76346160dbc05b156fb6864c58493b757d3c60: Status 404 returned error can't find the container with id 25cae02710136b6bf774260d9e76346160dbc05b156fb6864c58493b757d3c60
Sep 16 16:24:59.117204 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:59.117167    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk" event={"ID":"16dcb650-2717-47f9-bbd4-941bb2b6ba18","Type":"ContainerStarted","Data":"c4af48b2f2b15eb47f6037bdbccddce6ae44592bc6f8472f760540582b44b38c"}
Sep 16 16:24:59.117612 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:59.117228    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk" event={"ID":"16dcb650-2717-47f9-bbd4-941bb2b6ba18","Type":"ContainerStarted","Data":"25cae02710136b6bf774260d9e76346160dbc05b156fb6864c58493b757d3c60"}
Sep 16 16:24:59.135645 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:59.135594    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk" podStartSLOduration=2.13557811 podStartE2EDuration="2.13557811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:24:59.135013658 +0000 UTC m=+2095.654636434" watchObservedRunningTime="2026-09-16 16:24:59.13557811 +0000 UTC m=+2095.655200911"
Sep 16 16:24:59.286100 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:24:59.286055    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:25:07.769712 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:07.769678    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"]
Sep 16 16:25:07.774787 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:07.774764    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:07.777718 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:07.777698    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 16:25:07.785860 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:07.785792    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"]
Sep 16 16:25:07.964988 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:07.964953    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:07.965153 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:07.965000    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqfgx\" (UniqueName: \"kubernetes.io/projected/303530c5-c9c3-45fd-83f4-8ace78b9da46-kube-api-access-cqfgx\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:07.965153 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:07.965059    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:07.965153 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:07.965091    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303530c5-c9c3-45fd-83f4-8ace78b9da46-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:07.965153 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:07.965122    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:07.965335 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:07.965203    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-home\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:08.066012 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.065934    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-home\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:08.066012 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.065990    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:08.066218 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.066022    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqfgx\" (UniqueName: \"kubernetes.io/projected/303530c5-c9c3-45fd-83f4-8ace78b9da46-kube-api-access-cqfgx\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:08.066218 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.066041    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:08.066218 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.066055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303530c5-c9c3-45fd-83f4-8ace78b9da46-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:08.066218 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.066077    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:08.066420 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.066383    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:08.066420 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.066387    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-home\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:08.066492 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.066461    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:08.068311 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.068290    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:08.068507 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.068488    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303530c5-c9c3-45fd-83f4-8ace78b9da46-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:08.082883 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.082859    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqfgx\" (UniqueName: \"kubernetes.io/projected/303530c5-c9c3-45fd-83f4-8ace78b9da46-kube-api-access-cqfgx\") pod \"test-div-v2-kserve-5f6fdb6b77-sncch\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:08.085623 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.085606    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:08.285828 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.285789    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:25:08.298634 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.298605    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:25:08.352564 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:08.352536    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"]
Sep 16 16:25:08.355183 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:25:08.355153    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303530c5_c9c3_45fd_83f4_8ace78b9da46.slice/crio-277a17cdf8e558d4bf4672713d74640fb3e263f2b52857ec075a5c832561297b WatchSource:0}: Error finding container 277a17cdf8e558d4bf4672713d74640fb3e263f2b52857ec075a5c832561297b: Status 404 returned error can't find the container with id 277a17cdf8e558d4bf4672713d74640fb3e263f2b52857ec075a5c832561297b
Sep 16 16:25:09.169582 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:09.169545    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch" event={"ID":"303530c5-c9c3-45fd-83f4-8ace78b9da46","Type":"ContainerStarted","Data":"8b925fee9cc949cb7c17805ac458180837cb07b84cb37975565f72c46687d5cf"}
Sep 16 16:25:09.169958 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:09.169601    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch" event={"ID":"303530c5-c9c3-45fd-83f4-8ace78b9da46","Type":"ContainerStarted","Data":"277a17cdf8e558d4bf4672713d74640fb3e263f2b52857ec075a5c832561297b"}
Sep 16 16:25:09.180373 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:09.180349    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:25:09.186128 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:09.186089    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch" podStartSLOduration=2.186054793 podStartE2EDuration="2.186054793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:25:09.185471906 +0000 UTC m=+2105.705094708" watchObservedRunningTime="2026-09-16 16:25:09.186054793 +0000 UTC m=+2105.705677568"
Sep 16 16:25:10.086034 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:10.086002    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:18.086578 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:18.086544    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:18.098465 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:18.098440    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:18.225290 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:18.225263    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:38.190047 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:38.190010    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"]
Sep 16 16:25:38.476644 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:38.476562    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"]
Sep 16 16:25:53.208672 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.208609    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk" podUID="16dcb650-2717-47f9-bbd4-941bb2b6ba18" containerName="main" containerID="cri-o://c4af48b2f2b15eb47f6037bdbccddce6ae44592bc6f8472f760540582b44b38c" gracePeriod=45
Sep 16 16:25:53.381975 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.381939    2574 generic.go:356] "Generic (PLEG): container finished" podID="16dcb650-2717-47f9-bbd4-941bb2b6ba18" containerID="c4af48b2f2b15eb47f6037bdbccddce6ae44592bc6f8472f760540582b44b38c" exitCode=0
Sep 16 16:25:53.382112 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.382016    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk" event={"ID":"16dcb650-2717-47f9-bbd4-941bb2b6ba18","Type":"ContainerDied","Data":"c4af48b2f2b15eb47f6037bdbccddce6ae44592bc6f8472f760540582b44b38c"}
Sep 16 16:25:53.460870 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.460811    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:25:53.494939 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.494901    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch" podUID="303530c5-c9c3-45fd-83f4-8ace78b9da46" containerName="main" containerID="cri-o://8b925fee9cc949cb7c17805ac458180837cb07b84cb37975565f72c46687d5cf" gracePeriod=45
Sep 16 16:25:53.566013 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.565985    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-tmp-dir\") pod \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") "
Sep 16 16:25:53.566128 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.566080    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-dshm\") pod \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") "
Sep 16 16:25:53.566128 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.566097    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-home\") pod \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") "
Sep 16 16:25:53.566214 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.566161    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16dcb650-2717-47f9-bbd4-941bb2b6ba18-tls-certs\") pod \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") "
Sep 16 16:25:53.566214 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.566182    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5fbr\" (UniqueName: \"kubernetes.io/projected/16dcb650-2717-47f9-bbd4-941bb2b6ba18-kube-api-access-m5fbr\") pod \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") "
Sep 16 16:25:53.566214 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.566211    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-model-cache\") pod \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\" (UID: \"16dcb650-2717-47f9-bbd4-941bb2b6ba18\") "
Sep 16 16:25:53.566369 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.566222    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16dcb650-2717-47f9-bbd4-941bb2b6ba18" (UID: "16dcb650-2717-47f9-bbd4-941bb2b6ba18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:25:53.566425 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.566401    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-home" (OuterVolumeSpecName: "home") pod "16dcb650-2717-47f9-bbd4-941bb2b6ba18" (UID: "16dcb650-2717-47f9-bbd4-941bb2b6ba18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:25:53.566425 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.566417    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:25:53.566615 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.566591    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-model-cache" (OuterVolumeSpecName: "model-cache") pod "16dcb650-2717-47f9-bbd4-941bb2b6ba18" (UID: "16dcb650-2717-47f9-bbd4-941bb2b6ba18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:25:53.568275 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.568247    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-dshm" (OuterVolumeSpecName: "dshm") pod "16dcb650-2717-47f9-bbd4-941bb2b6ba18" (UID: "16dcb650-2717-47f9-bbd4-941bb2b6ba18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:25:53.568354 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.568252    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16dcb650-2717-47f9-bbd4-941bb2b6ba18-kube-api-access-m5fbr" (OuterVolumeSpecName: "kube-api-access-m5fbr") pod "16dcb650-2717-47f9-bbd4-941bb2b6ba18" (UID: "16dcb650-2717-47f9-bbd4-941bb2b6ba18"). InnerVolumeSpecName "kube-api-access-m5fbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:25:53.568354 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.568302    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16dcb650-2717-47f9-bbd4-941bb2b6ba18-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16dcb650-2717-47f9-bbd4-941bb2b6ba18" (UID: "16dcb650-2717-47f9-bbd4-941bb2b6ba18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:25:53.667356 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.667322    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:25:53.667513 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.667361    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:25:53.667513 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.667375    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16dcb650-2717-47f9-bbd4-941bb2b6ba18-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:25:53.667513 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.667386    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16dcb650-2717-47f9-bbd4-941bb2b6ba18-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:25:53.667513 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.667399    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5fbr\" (UniqueName: \"kubernetes.io/projected/16dcb650-2717-47f9-bbd4-941bb2b6ba18-kube-api-access-m5fbr\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:25:53.731439 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.731414    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:53.868870 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.868838    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-dshm\") pod \"303530c5-c9c3-45fd-83f4-8ace78b9da46\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") "
Sep 16 16:25:53.869005 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.868893    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-model-cache\") pod \"303530c5-c9c3-45fd-83f4-8ace78b9da46\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") "
Sep 16 16:25:53.869005 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.868970    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqfgx\" (UniqueName: \"kubernetes.io/projected/303530c5-c9c3-45fd-83f4-8ace78b9da46-kube-api-access-cqfgx\") pod \"303530c5-c9c3-45fd-83f4-8ace78b9da46\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") "
Sep 16 16:25:53.869005 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.868999    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-home\") pod \"303530c5-c9c3-45fd-83f4-8ace78b9da46\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") "
Sep 16 16:25:53.869141 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.869023    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-tmp-dir\") pod \"303530c5-c9c3-45fd-83f4-8ace78b9da46\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") "
Sep 16 16:25:53.869141 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.869048    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303530c5-c9c3-45fd-83f4-8ace78b9da46-tls-certs\") pod \"303530c5-c9c3-45fd-83f4-8ace78b9da46\" (UID: \"303530c5-c9c3-45fd-83f4-8ace78b9da46\") "
Sep 16 16:25:53.869268 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.869240    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-home" (OuterVolumeSpecName: "home") pod "303530c5-c9c3-45fd-83f4-8ace78b9da46" (UID: "303530c5-c9c3-45fd-83f4-8ace78b9da46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:25:53.869268 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.869235    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-model-cache" (OuterVolumeSpecName: "model-cache") pod "303530c5-c9c3-45fd-83f4-8ace78b9da46" (UID: "303530c5-c9c3-45fd-83f4-8ace78b9da46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:25:53.869268 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.869257    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "303530c5-c9c3-45fd-83f4-8ace78b9da46" (UID: "303530c5-c9c3-45fd-83f4-8ace78b9da46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:25:53.869421 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.869352    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:25:53.869421 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.869368    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:25:53.869421 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.869382    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:25:53.871048 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.871019    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-dshm" (OuterVolumeSpecName: "dshm") pod "303530c5-c9c3-45fd-83f4-8ace78b9da46" (UID: "303530c5-c9c3-45fd-83f4-8ace78b9da46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:25:53.871048 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.871043    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303530c5-c9c3-45fd-83f4-8ace78b9da46-kube-api-access-cqfgx" (OuterVolumeSpecName: "kube-api-access-cqfgx") pod "303530c5-c9c3-45fd-83f4-8ace78b9da46" (UID: "303530c5-c9c3-45fd-83f4-8ace78b9da46"). InnerVolumeSpecName "kube-api-access-cqfgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:25:53.871183 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.871022    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/303530c5-c9c3-45fd-83f4-8ace78b9da46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "303530c5-c9c3-45fd-83f4-8ace78b9da46" (UID: "303530c5-c9c3-45fd-83f4-8ace78b9da46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:25:53.970680 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.970654    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/303530c5-c9c3-45fd-83f4-8ace78b9da46-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:25:53.970680 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.970676    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/303530c5-c9c3-45fd-83f4-8ace78b9da46-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:25:53.970816 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:53.970686    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqfgx\" (UniqueName: \"kubernetes.io/projected/303530c5-c9c3-45fd-83f4-8ace78b9da46-kube-api-access-cqfgx\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:25:54.391734 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:54.391694    2574 generic.go:356] "Generic (PLEG): container finished" podID="303530c5-c9c3-45fd-83f4-8ace78b9da46" containerID="8b925fee9cc949cb7c17805ac458180837cb07b84cb37975565f72c46687d5cf" exitCode=0
Sep 16 16:25:54.392181 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:54.391748    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch" event={"ID":"303530c5-c9c3-45fd-83f4-8ace78b9da46","Type":"ContainerDied","Data":"8b925fee9cc949cb7c17805ac458180837cb07b84cb37975565f72c46687d5cf"}
Sep 16 16:25:54.392181 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:54.391760    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"
Sep 16 16:25:54.392181 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:54.391816    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch" event={"ID":"303530c5-c9c3-45fd-83f4-8ace78b9da46","Type":"ContainerDied","Data":"277a17cdf8e558d4bf4672713d74640fb3e263f2b52857ec075a5c832561297b"}
Sep 16 16:25:54.392181 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:54.391836    2574 scope.go:117] "RemoveContainer" containerID="8b925fee9cc949cb7c17805ac458180837cb07b84cb37975565f72c46687d5cf"
Sep 16 16:25:54.393905 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:54.393883    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk" event={"ID":"16dcb650-2717-47f9-bbd4-941bb2b6ba18","Type":"ContainerDied","Data":"25cae02710136b6bf774260d9e76346160dbc05b156fb6864c58493b757d3c60"}
Sep 16 16:25:54.394014 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:54.393932    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"
Sep 16 16:25:54.403815 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:54.403797    2574 scope.go:117] "RemoveContainer" containerID="8b925fee9cc949cb7c17805ac458180837cb07b84cb37975565f72c46687d5cf"
Sep 16 16:25:54.404154 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:25:54.404130    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b925fee9cc949cb7c17805ac458180837cb07b84cb37975565f72c46687d5cf\": container with ID starting with 8b925fee9cc949cb7c17805ac458180837cb07b84cb37975565f72c46687d5cf not found: ID does not exist" containerID="8b925fee9cc949cb7c17805ac458180837cb07b84cb37975565f72c46687d5cf"
Sep 16 16:25:54.404211 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:54.404165    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b925fee9cc949cb7c17805ac458180837cb07b84cb37975565f72c46687d5cf"} err="failed to get container status \"8b925fee9cc949cb7c17805ac458180837cb07b84cb37975565f72c46687d5cf\": rpc error: code = NotFound desc = could not find container \"8b925fee9cc949cb7c17805ac458180837cb07b84cb37975565f72c46687d5cf\": container with ID starting with 8b925fee9cc949cb7c17805ac458180837cb07b84cb37975565f72c46687d5cf not found: ID does not exist"
Sep 16 16:25:54.404211 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:54.404189    2574 scope.go:117] "RemoveContainer" containerID="c4af48b2f2b15eb47f6037bdbccddce6ae44592bc6f8472f760540582b44b38c"
Sep 16 16:25:54.410835 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:54.410818    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"]
Sep 16 16:25:54.416010 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:54.415985    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kphtk"]
Sep 16 16:25:54.428268 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:54.428245    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"]
Sep 16 16:25:54.430980 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:54.430960    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-sncch"]
Sep 16 16:25:55.994328 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:55.994256    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16dcb650-2717-47f9-bbd4-941bb2b6ba18" path="/var/lib/kubelet/pods/16dcb650-2717-47f9-bbd4-941bb2b6ba18/volumes"
Sep 16 16:25:55.994719 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:25:55.994702    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="303530c5-c9c3-45fd-83f4-8ace78b9da46" path="/var/lib/kubelet/pods/303530c5-c9c3-45fd-83f4-8ace78b9da46/volumes"
Sep 16 16:26:14.277060 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.277028    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"]
Sep 16 16:26:14.277532 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.277427    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="303530c5-c9c3-45fd-83f4-8ace78b9da46" containerName="main"
Sep 16 16:26:14.277532 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.277443    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="303530c5-c9c3-45fd-83f4-8ace78b9da46" containerName="main"
Sep 16 16:26:14.277532 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.277463    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16dcb650-2717-47f9-bbd4-941bb2b6ba18" containerName="main"
Sep 16 16:26:14.277532 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.277470    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="16dcb650-2717-47f9-bbd4-941bb2b6ba18" containerName="main"
Sep 16 16:26:14.277709 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.277575    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="16dcb650-2717-47f9-bbd4-941bb2b6ba18" containerName="main"
Sep 16 16:26:14.277709 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.277586    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="303530c5-c9c3-45fd-83f4-8ace78b9da46" containerName="main"
Sep 16 16:26:14.280898 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.280877    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.283643 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.283616    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 16:26:14.283759 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.283636    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:26:14.283759 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.283709    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 16:26:14.283881 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.283714    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-swnq8\""
Sep 16 16:26:14.284367 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.284344    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 16:26:14.284673 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.284652    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 16:26:14.292462 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.292442    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"]
Sep 16 16:26:14.350086 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.350058    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39d427ad-9f60-41c5-a538-de64dcad6bc2-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.350086 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.350089    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjwgg\" (UniqueName: \"kubernetes.io/projected/39d427ad-9f60-41c5-a538-de64dcad6bc2-kube-api-access-tjwgg\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.350279 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.350115    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.350279 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.350171    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.350279 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.350209    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-model-cache\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.350279 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.350254    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39d427ad-9f60-41c5-a538-de64dcad6bc2-tls-certs\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.350407 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.350320    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-dshm\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.350407 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.350349    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-home\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.451681 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.451650    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39d427ad-9f60-41c5-a538-de64dcad6bc2-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.451681 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.451680    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjwgg\" (UniqueName: \"kubernetes.io/projected/39d427ad-9f60-41c5-a538-de64dcad6bc2-kube-api-access-tjwgg\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.451908 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.451704    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.451908 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.451727    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.451908 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.451744    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-model-cache\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.451908 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.451761    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39d427ad-9f60-41c5-a538-de64dcad6bc2-tls-certs\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.451908 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.451808    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-dshm\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.451908 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.451831    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-home\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.452196 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.452123    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.452196 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.452142    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-model-cache\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.452273 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.452231    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-home\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.452310 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.452279    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.452451 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.452434    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39d427ad-9f60-41c5-a538-de64dcad6bc2-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.454028 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.454005    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-dshm\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.454269 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.454251    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39d427ad-9f60-41c5-a538-de64dcad6bc2-tls-certs\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.459104 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.459077    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjwgg\" (UniqueName: \"kubernetes.io/projected/39d427ad-9f60-41c5-a538-de64dcad6bc2-kube-api-access-tjwgg\") pod \"tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.511444 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.511410    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"]
Sep 16 16:26:14.516112 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.516092    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:14.529847 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.529785    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"]
Sep 16 16:26:14.552584 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.552554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f825fb-3464-4e45-a2c4-52d4d69ca602-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:14.552732 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.552604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f825fb-3464-4e45-a2c4-52d4d69ca602-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:14.552732 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.552677    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f825fb-3464-4e45-a2c4-52d4d69ca602-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:14.552732 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.552725    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f825fb-3464-4e45-a2c4-52d4d69ca602-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:14.591752 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.591721    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:14.655107 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.654294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f825fb-3464-4e45-a2c4-52d4d69ca602-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:14.655107 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.654388    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f825fb-3464-4e45-a2c4-52d4d69ca602-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:14.655107 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.654441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f825fb-3464-4e45-a2c4-52d4d69ca602-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:14.655107 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.654498    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f825fb-3464-4e45-a2c4-52d4d69ca602-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:14.655107 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.654778    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f825fb-3464-4e45-a2c4-52d4d69ca602-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:14.655107 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.654921    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f825fb-3464-4e45-a2c4-52d4d69ca602-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:14.655493 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.655265    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f825fb-3464-4e45-a2c4-52d4d69ca602-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:14.658145 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.658084    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f825fb-3464-4e45-a2c4-52d4d69ca602-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:14.828533 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.828428    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:14.869208 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:14.869172    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"]
Sep 16 16:26:14.872652 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:26:14.872627    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d427ad_9f60_41c5_a538_de64dcad6bc2.slice/crio-0ed9cba800345b25513b21cf9a55934f2ccca155b90c60992e7ab01d744c23a5 WatchSource:0}: Error finding container 0ed9cba800345b25513b21cf9a55934f2ccca155b90c60992e7ab01d744c23a5: Status 404 returned error can't find the container with id 0ed9cba800345b25513b21cf9a55934f2ccca155b90c60992e7ab01d744c23a5
Sep 16 16:26:15.102684 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:15.102659    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"]
Sep 16 16:26:15.104830 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:26:15.104802    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f825fb_3464_4e45_a2c4_52d4d69ca602.slice/crio-43a1d69409c9d792750d456d0607b4f7179f28a09176787c1af66550159f364f WatchSource:0}: Error finding container 43a1d69409c9d792750d456d0607b4f7179f28a09176787c1af66550159f364f: Status 404 returned error can't find the container with id 43a1d69409c9d792750d456d0607b4f7179f28a09176787c1af66550159f364f
Sep 16 16:26:15.495079 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:15.494984    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp" event={"ID":"39d427ad-9f60-41c5-a538-de64dcad6bc2","Type":"ContainerStarted","Data":"339e1c0772f988266dfa6e517a27fe7af4dc03f99187849f5e0cc6c9d188c8ef"}
Sep 16 16:26:15.495079 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:15.495060    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp" event={"ID":"39d427ad-9f60-41c5-a538-de64dcad6bc2","Type":"ContainerStarted","Data":"0ed9cba800345b25513b21cf9a55934f2ccca155b90c60992e7ab01d744c23a5"}
Sep 16 16:26:15.496817 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:15.496787    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44" event={"ID":"37f825fb-3464-4e45-a2c4-52d4d69ca602","Type":"ContainerStarted","Data":"9ce792c9a6e680fddccea201f2949ebda4bfae82da37fe07a0828d486ca7896d"}
Sep 16 16:26:15.496906 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:15.496843    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44" event={"ID":"37f825fb-3464-4e45-a2c4-52d4d69ca602","Type":"ContainerStarted","Data":"43a1d69409c9d792750d456d0607b4f7179f28a09176787c1af66550159f364f"}
Sep 16 16:26:17.510804 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:17.510765    2574 generic.go:356] "Generic (PLEG): container finished" podID="39d427ad-9f60-41c5-a538-de64dcad6bc2" containerID="339e1c0772f988266dfa6e517a27fe7af4dc03f99187849f5e0cc6c9d188c8ef" exitCode=0
Sep 16 16:26:17.511289 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:17.510819    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp" event={"ID":"39d427ad-9f60-41c5-a538-de64dcad6bc2","Type":"ContainerDied","Data":"339e1c0772f988266dfa6e517a27fe7af4dc03f99187849f5e0cc6c9d188c8ef"}
Sep 16 16:26:18.517578 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:18.517535    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp" event={"ID":"39d427ad-9f60-41c5-a538-de64dcad6bc2","Type":"ContainerStarted","Data":"ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857"}
Sep 16 16:26:18.519125 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:18.519098    2574 generic.go:356] "Generic (PLEG): container finished" podID="37f825fb-3464-4e45-a2c4-52d4d69ca602" containerID="9ce792c9a6e680fddccea201f2949ebda4bfae82da37fe07a0828d486ca7896d" exitCode=0
Sep 16 16:26:18.519247 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:18.519140    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44" event={"ID":"37f825fb-3464-4e45-a2c4-52d4d69ca602","Type":"ContainerDied","Data":"9ce792c9a6e680fddccea201f2949ebda4bfae82da37fe07a0828d486ca7896d"}
Sep 16 16:26:18.538844 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:18.538802    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp" podStartSLOduration=2.050103477 podStartE2EDuration="4.538787263s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.488683786s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:26:18.537420338 +0000 UTC m=+2175.057043115" watchObservedRunningTime="2026-09-16 16:26:18.538787263 +0000 UTC m=+2175.058410039"
Sep 16 16:26:18.591893 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:18.591865    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:19.526585 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:19.526549    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44" event={"ID":"37f825fb-3464-4e45-a2c4-52d4d69ca602","Type":"ContainerStarted","Data":"3c7c2544e76dc59fb889268b31f02377d8bbae0e93c95a983aaa17e24cece421"}
Sep 16 16:26:19.546200 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:19.546154    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44" podStartSLOduration=3.034416414 podStartE2EDuration="5.546138973s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.511722559s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:26:19.543915009 +0000 UTC m=+2176.063537784" watchObservedRunningTime="2026-09-16 16:26:19.546138973 +0000 UTC m=+2176.065761748"
Sep 16 16:26:24.592414 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:24.592379    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:24.607035 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:24.607006    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:25.568669 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:25.568634    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:29.527760 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:29.527716    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:49.538583 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:49.538544    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:50.934640 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:50.934606    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44_37f825fb-3464-4e45-a2c4-52d4d69ca602/main/0.log"
Sep 16 16:26:51.306701 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.306606    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"]
Sep 16 16:26:51.310130 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.310092    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"]
Sep 16 16:26:51.310367 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.310294    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44" podUID="37f825fb-3464-4e45-a2c4-52d4d69ca602" containerName="main" containerID="cri-o://3c7c2544e76dc59fb889268b31f02377d8bbae0e93c95a983aaa17e24cece421" gracePeriod=30
Sep 16 16:26:51.319226 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:26:51.319192    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:26:51.319226 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:26:51.319226 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:26:51.319226 ip-10-0-143-143 kubenswrapper[2574]:  > containerID="ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857" cmd=["/bin/sleep","15"]
Sep 16 16:26:51.319442 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.319241    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp" message="" err=<
Sep 16 16:26:51.319442 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:26:51.319442 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:26:51.319442 ip-10-0-143-143 kubenswrapper[2574]:  >
Sep 16 16:26:51.319442 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:26:51.319252    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:26:51.319442 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:26:51.319442 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:26:51.319442 ip-10-0-143-143 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp" podUID="39d427ad-9f60-41c5-a538-de64dcad6bc2" containerName="main" containerID="cri-o://ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857"
Sep 16 16:26:51.319442 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.319293    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp" podUID="39d427ad-9f60-41c5-a538-de64dcad6bc2" containerName="main" containerID="cri-o://ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857" gracePeriod=60
Sep 16 16:26:51.557737 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.557643    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp" podUID="39d427ad-9f60-41c5-a538-de64dcad6bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 16 16:26:51.682328 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.682299    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:51.691993 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.691965    2574 generic.go:356] "Generic (PLEG): container finished" podID="39d427ad-9f60-41c5-a538-de64dcad6bc2" containerID="ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857" exitCode=0
Sep 16 16:26:51.692118 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.692057    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"
Sep 16 16:26:51.692118 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.692072    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp" event={"ID":"39d427ad-9f60-41c5-a538-de64dcad6bc2","Type":"ContainerDied","Data":"ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857"}
Sep 16 16:26:51.692195 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.692129    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp" event={"ID":"39d427ad-9f60-41c5-a538-de64dcad6bc2","Type":"ContainerDied","Data":"0ed9cba800345b25513b21cf9a55934f2ccca155b90c60992e7ab01d744c23a5"}
Sep 16 16:26:51.692195 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.692174    2574 scope.go:117] "RemoveContainer" containerID="ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857"
Sep 16 16:26:51.702102 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.702077    2574 scope.go:117] "RemoveContainer" containerID="339e1c0772f988266dfa6e517a27fe7af4dc03f99187849f5e0cc6c9d188c8ef"
Sep 16 16:26:51.712244 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.712220    2574 scope.go:117] "RemoveContainer" containerID="ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857"
Sep 16 16:26:51.712638 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:26:51.712600    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857\": container with ID starting with ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857 not found: ID does not exist" containerID="ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857"
Sep 16 16:26:51.712740 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.712647    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857"} err="failed to get container status \"ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857\": rpc error: code = NotFound desc = could not find container \"ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857\": container with ID starting with ffad876c864176671f06d9a31605a5ae19c3d0ebc96a4c473b70377cc5efa857 not found: ID does not exist"
Sep 16 16:26:51.712740 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.712672    2574 scope.go:117] "RemoveContainer" containerID="339e1c0772f988266dfa6e517a27fe7af4dc03f99187849f5e0cc6c9d188c8ef"
Sep 16 16:26:51.712996 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:26:51.712969    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339e1c0772f988266dfa6e517a27fe7af4dc03f99187849f5e0cc6c9d188c8ef\": container with ID starting with 339e1c0772f988266dfa6e517a27fe7af4dc03f99187849f5e0cc6c9d188c8ef not found: ID does not exist" containerID="339e1c0772f988266dfa6e517a27fe7af4dc03f99187849f5e0cc6c9d188c8ef"
Sep 16 16:26:51.713064 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.712993    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339e1c0772f988266dfa6e517a27fe7af4dc03f99187849f5e0cc6c9d188c8ef"} err="failed to get container status \"339e1c0772f988266dfa6e517a27fe7af4dc03f99187849f5e0cc6c9d188c8ef\": rpc error: code = NotFound desc = could not find container \"339e1c0772f988266dfa6e517a27fe7af4dc03f99187849f5e0cc6c9d188c8ef\": container with ID starting with 339e1c0772f988266dfa6e517a27fe7af4dc03f99187849f5e0cc6c9d188c8ef not found: ID does not exist"
Sep 16 16:26:51.715274 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.715254    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjwgg\" (UniqueName: \"kubernetes.io/projected/39d427ad-9f60-41c5-a538-de64dcad6bc2-kube-api-access-tjwgg\") pod \"39d427ad-9f60-41c5-a538-de64dcad6bc2\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") "
Sep 16 16:26:51.715355 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.715338    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-home\") pod \"39d427ad-9f60-41c5-a538-de64dcad6bc2\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") "
Sep 16 16:26:51.715410 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.715378    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-model-cache\") pod \"39d427ad-9f60-41c5-a538-de64dcad6bc2\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") "
Sep 16 16:26:51.715464 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.715418    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-tmp-dir\") pod \"39d427ad-9f60-41c5-a538-de64dcad6bc2\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") "
Sep 16 16:26:51.715532 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.715467    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39d427ad-9f60-41c5-a538-de64dcad6bc2-tls-certs\") pod \"39d427ad-9f60-41c5-a538-de64dcad6bc2\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") "
Sep 16 16:26:51.715532 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.715496    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-kserve-provision-location\") pod \"39d427ad-9f60-41c5-a538-de64dcad6bc2\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") "
Sep 16 16:26:51.715656 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.715571    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39d427ad-9f60-41c5-a538-de64dcad6bc2-cabundle-cert\") pod \"39d427ad-9f60-41c5-a538-de64dcad6bc2\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") "
Sep 16 16:26:51.715656 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.715587    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-home" (OuterVolumeSpecName: "home") pod "39d427ad-9f60-41c5-a538-de64dcad6bc2" (UID: "39d427ad-9f60-41c5-a538-de64dcad6bc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:26:51.715656 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.715601    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-dshm\") pod \"39d427ad-9f60-41c5-a538-de64dcad6bc2\" (UID: \"39d427ad-9f60-41c5-a538-de64dcad6bc2\") "
Sep 16 16:26:51.715656 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.715613    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-model-cache" (OuterVolumeSpecName: "model-cache") pod "39d427ad-9f60-41c5-a538-de64dcad6bc2" (UID: "39d427ad-9f60-41c5-a538-de64dcad6bc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:26:51.715853 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.715742    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39d427ad-9f60-41c5-a538-de64dcad6bc2" (UID: "39d427ad-9f60-41c5-a538-de64dcad6bc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:26:51.715980 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.715961    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.716045 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.715983    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.716045 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.715994    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.716146 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.716106    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d427ad-9f60-41c5-a538-de64dcad6bc2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "39d427ad-9f60-41c5-a538-de64dcad6bc2" (UID: "39d427ad-9f60-41c5-a538-de64dcad6bc2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:26:51.718044 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.718023    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d427ad-9f60-41c5-a538-de64dcad6bc2-kube-api-access-tjwgg" (OuterVolumeSpecName: "kube-api-access-tjwgg") pod "39d427ad-9f60-41c5-a538-de64dcad6bc2" (UID: "39d427ad-9f60-41c5-a538-de64dcad6bc2"). InnerVolumeSpecName "kube-api-access-tjwgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:26:51.718136 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.718023    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39d427ad-9f60-41c5-a538-de64dcad6bc2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39d427ad-9f60-41c5-a538-de64dcad6bc2" (UID: "39d427ad-9f60-41c5-a538-de64dcad6bc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:26:51.718444 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.718422    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-dshm" (OuterVolumeSpecName: "dshm") pod "39d427ad-9f60-41c5-a538-de64dcad6bc2" (UID: "39d427ad-9f60-41c5-a538-de64dcad6bc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:26:51.738189 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.738158    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39d427ad-9f60-41c5-a538-de64dcad6bc2" (UID: "39d427ad-9f60-41c5-a538-de64dcad6bc2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:26:51.816949 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.816873    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39d427ad-9f60-41c5-a538-de64dcad6bc2-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.816949 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.816904    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.816949 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.816915    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39d427ad-9f60-41c5-a538-de64dcad6bc2-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.816949 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.816924    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39d427ad-9f60-41c5-a538-de64dcad6bc2-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.816949 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.816932    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjwgg\" (UniqueName: \"kubernetes.io/projected/39d427ad-9f60-41c5-a538-de64dcad6bc2-kube-api-access-tjwgg\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:26:51.948844 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:51.948816    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:52.015898 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.015866    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"]
Sep 16 16:26:52.018795 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.018770    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75c8c5f496-xkhnp"]
Sep 16 16:26:52.018992 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.018973    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f825fb-3464-4e45-a2c4-52d4d69ca602-tls-certs\") pod \"37f825fb-3464-4e45-a2c4-52d4d69ca602\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") "
Sep 16 16:26:52.019048 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.019006    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f825fb-3464-4e45-a2c4-52d4d69ca602-tmp-dir\") pod \"37f825fb-3464-4e45-a2c4-52d4d69ca602\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") "
Sep 16 16:26:52.019048 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.019027    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f825fb-3464-4e45-a2c4-52d4d69ca602-cabundle-cert\") pod \"37f825fb-3464-4e45-a2c4-52d4d69ca602\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") "
Sep 16 16:26:52.019157 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.019078    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f825fb-3464-4e45-a2c4-52d4d69ca602-kserve-provision-location\") pod \"37f825fb-3464-4e45-a2c4-52d4d69ca602\" (UID: \"37f825fb-3464-4e45-a2c4-52d4d69ca602\") "
Sep 16 16:26:52.019406 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.019376    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f825fb-3464-4e45-a2c4-52d4d69ca602-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37f825fb-3464-4e45-a2c4-52d4d69ca602" (UID: "37f825fb-3464-4e45-a2c4-52d4d69ca602"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:26:52.019406 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.019403    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f825fb-3464-4e45-a2c4-52d4d69ca602-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37f825fb-3464-4e45-a2c4-52d4d69ca602" (UID: "37f825fb-3464-4e45-a2c4-52d4d69ca602"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:26:52.021128 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.021104    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f825fb-3464-4e45-a2c4-52d4d69ca602-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37f825fb-3464-4e45-a2c4-52d4d69ca602" (UID: "37f825fb-3464-4e45-a2c4-52d4d69ca602"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:26:52.040876 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.040847    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f825fb-3464-4e45-a2c4-52d4d69ca602-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37f825fb-3464-4e45-a2c4-52d4d69ca602" (UID: "37f825fb-3464-4e45-a2c4-52d4d69ca602"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:26:52.119772 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.119747    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37f825fb-3464-4e45-a2c4-52d4d69ca602-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:26:52.119772 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.119772    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f825fb-3464-4e45-a2c4-52d4d69ca602-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:26:52.119959 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.119781    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f825fb-3464-4e45-a2c4-52d4d69ca602-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:26:52.119959 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.119789    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37f825fb-3464-4e45-a2c4-52d4d69ca602-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:26:52.698504 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.698423    2574 generic.go:356] "Generic (PLEG): container finished" podID="37f825fb-3464-4e45-a2c4-52d4d69ca602" containerID="3c7c2544e76dc59fb889268b31f02377d8bbae0e93c95a983aaa17e24cece421" exitCode=0
Sep 16 16:26:52.698504 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.698463    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"
Sep 16 16:26:52.698730 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.698504    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44" event={"ID":"37f825fb-3464-4e45-a2c4-52d4d69ca602","Type":"ContainerDied","Data":"3c7c2544e76dc59fb889268b31f02377d8bbae0e93c95a983aaa17e24cece421"}
Sep 16 16:26:52.698730 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.698583    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44" event={"ID":"37f825fb-3464-4e45-a2c4-52d4d69ca602","Type":"ContainerDied","Data":"43a1d69409c9d792750d456d0607b4f7179f28a09176787c1af66550159f364f"}
Sep 16 16:26:52.698730 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.698614    2574 scope.go:117] "RemoveContainer" containerID="3c7c2544e76dc59fb889268b31f02377d8bbae0e93c95a983aaa17e24cece421"
Sep 16 16:26:52.709815 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.709798    2574 scope.go:117] "RemoveContainer" containerID="9ce792c9a6e680fddccea201f2949ebda4bfae82da37fe07a0828d486ca7896d"
Sep 16 16:26:52.717257 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.717239    2574 scope.go:117] "RemoveContainer" containerID="3c7c2544e76dc59fb889268b31f02377d8bbae0e93c95a983aaa17e24cece421"
Sep 16 16:26:52.717509 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:26:52.717488    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7c2544e76dc59fb889268b31f02377d8bbae0e93c95a983aaa17e24cece421\": container with ID starting with 3c7c2544e76dc59fb889268b31f02377d8bbae0e93c95a983aaa17e24cece421 not found: ID does not exist" containerID="3c7c2544e76dc59fb889268b31f02377d8bbae0e93c95a983aaa17e24cece421"
Sep 16 16:26:52.717732 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.717536    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7c2544e76dc59fb889268b31f02377d8bbae0e93c95a983aaa17e24cece421"} err="failed to get container status \"3c7c2544e76dc59fb889268b31f02377d8bbae0e93c95a983aaa17e24cece421\": rpc error: code = NotFound desc = could not find container \"3c7c2544e76dc59fb889268b31f02377d8bbae0e93c95a983aaa17e24cece421\": container with ID starting with 3c7c2544e76dc59fb889268b31f02377d8bbae0e93c95a983aaa17e24cece421 not found: ID does not exist"
Sep 16 16:26:52.717732 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.717554    2574 scope.go:117] "RemoveContainer" containerID="9ce792c9a6e680fddccea201f2949ebda4bfae82da37fe07a0828d486ca7896d"
Sep 16 16:26:52.717834 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:26:52.717778    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce792c9a6e680fddccea201f2949ebda4bfae82da37fe07a0828d486ca7896d\": container with ID starting with 9ce792c9a6e680fddccea201f2949ebda4bfae82da37fe07a0828d486ca7896d not found: ID does not exist" containerID="9ce792c9a6e680fddccea201f2949ebda4bfae82da37fe07a0828d486ca7896d"
Sep 16 16:26:52.717834 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.717796    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce792c9a6e680fddccea201f2949ebda4bfae82da37fe07a0828d486ca7896d"} err="failed to get container status \"9ce792c9a6e680fddccea201f2949ebda4bfae82da37fe07a0828d486ca7896d\": rpc error: code = NotFound desc = could not find container \"9ce792c9a6e680fddccea201f2949ebda4bfae82da37fe07a0828d486ca7896d\": container with ID starting with 9ce792c9a6e680fddccea201f2949ebda4bfae82da37fe07a0828d486ca7896d not found: ID does not exist"
Sep 16 16:26:52.720830 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.720810    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"]
Sep 16 16:26:52.726332 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:52.726313    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-798999b9f8-mfv44"]
Sep 16 16:26:53.994432 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:53.994397    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f825fb-3464-4e45-a2c4-52d4d69ca602" path="/var/lib/kubelet/pods/37f825fb-3464-4e45-a2c4-52d4d69ca602/volumes"
Sep 16 16:26:53.994899 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:26:53.994884    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39d427ad-9f60-41c5-a538-de64dcad6bc2" path="/var/lib/kubelet/pods/39d427ad-9f60-41c5-a538-de64dcad6bc2/volumes"
Sep 16 16:27:04.247632 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:04.247595    2574 scope.go:117] "RemoveContainer" containerID="eea2b4e4cea0ef104e65771a7bb194cad62fa7f221d75e80d9667d16d392abd6"
Sep 16 16:27:28.474644 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.474566    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"]
Sep 16 16:27:28.475020 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.474954    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d427ad-9f60-41c5-a538-de64dcad6bc2" containerName="main"
Sep 16 16:27:28.475020 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.474965    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d427ad-9f60-41c5-a538-de64dcad6bc2" containerName="main"
Sep 16 16:27:28.475020 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.474977    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f825fb-3464-4e45-a2c4-52d4d69ca602" containerName="storage-initializer"
Sep 16 16:27:28.475020 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.474983    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f825fb-3464-4e45-a2c4-52d4d69ca602" containerName="storage-initializer"
Sep 16 16:27:28.475020 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.474998    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d427ad-9f60-41c5-a538-de64dcad6bc2" containerName="storage-initializer"
Sep 16 16:27:28.475020 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.475004    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d427ad-9f60-41c5-a538-de64dcad6bc2" containerName="storage-initializer"
Sep 16 16:27:28.475020 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.475012    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f825fb-3464-4e45-a2c4-52d4d69ca602" containerName="main"
Sep 16 16:27:28.475020 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.475017    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f825fb-3464-4e45-a2c4-52d4d69ca602" containerName="main"
Sep 16 16:27:28.475263 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.475068    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f825fb-3464-4e45-a2c4-52d4d69ca602" containerName="main"
Sep 16 16:27:28.475263 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.475081    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d427ad-9f60-41c5-a538-de64dcad6bc2" containerName="main"
Sep 16 16:27:28.478308 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.478289    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.481571 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.481546    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 16 16:27:28.481694 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.481590    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 16:27:28.481694 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.481665    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 16:27:28.481694 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.481682    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-6x7fj\""
Sep 16 16:27:28.488645 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.488624    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"]
Sep 16 16:27:28.629700 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.629665    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-dshm\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.629700 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.629700    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-model-cache\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.629906 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.629809    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-home\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.629906 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.629864    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.629906 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.629891    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.630015 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.629926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7bw7\" (UniqueName: \"kubernetes.io/projected/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-kube-api-access-x7bw7\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.730904 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.730821    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.730904 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.730863    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.731104 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.730984    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7bw7\" (UniqueName: \"kubernetes.io/projected/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-kube-api-access-x7bw7\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.731104 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.731029    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-dshm\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.731217 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.731139    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-model-cache\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.731278 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.731224    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.731278 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.731228    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-home\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.731536 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.731502    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-model-cache\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.731641 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.731540    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-home\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.733076 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.733049    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-dshm\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.733439 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.733418    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.738718 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.738692    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7bw7\" (UniqueName: \"kubernetes.io/projected/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-kube-api-access-x7bw7\") pod \"decom-v1-kserve-6cb797fb75-b6jmf\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:28.788669 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:28.788647    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:29.083610 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:29.083581    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"]
Sep 16 16:27:29.085872 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:27:29.085834    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2cf8e6a_0e6f_498b_8e56_224469d47b5d.slice/crio-cdca3b66e78e4bb81cf4dc76ca92e50f42c5f774410114a25e6e3dcc88b62b31 WatchSource:0}: Error finding container cdca3b66e78e4bb81cf4dc76ca92e50f42c5f774410114a25e6e3dcc88b62b31: Status 404 returned error can't find the container with id cdca3b66e78e4bb81cf4dc76ca92e50f42c5f774410114a25e6e3dcc88b62b31
Sep 16 16:27:29.875095 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:29.875058    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf" event={"ID":"a2cf8e6a-0e6f-498b-8e56-224469d47b5d","Type":"ContainerStarted","Data":"670c26a6ca180ebd9b0cce823e7934a53b7512533ce94016c5bf6d20d2bc2604"}
Sep 16 16:27:29.875453 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:29.875113    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf" event={"ID":"a2cf8e6a-0e6f-498b-8e56-224469d47b5d","Type":"ContainerStarted","Data":"cdca3b66e78e4bb81cf4dc76ca92e50f42c5f774410114a25e6e3dcc88b62b31"}
Sep 16 16:27:29.892631 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:29.892578    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf" podStartSLOduration=1.892563241 podStartE2EDuration="1.892563241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:27:29.890346433 +0000 UTC m=+2246.409969208" watchObservedRunningTime="2026-09-16 16:27:29.892563241 +0000 UTC m=+2246.412186016"
Sep 16 16:27:30.789324 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:30.789293    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:38.789768 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:38.789728    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:38.801777 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:38.801752    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:27:38.929999 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:27:38.929968    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:28:06.705151 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.705113    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"]
Sep 16 16:28:06.708936 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.708919    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.711568 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.711492    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:28:06.712540 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.712359    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 16:28:06.712540 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.712425    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 16:28:06.712540 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.712437    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 16:28:06.712540 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.712424    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 16:28:06.712540 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.712431    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-qqx6s\""
Sep 16 16:28:06.718320 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.718300    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"]
Sep 16 16:28:06.855264 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.855234    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.855264 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.855267    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc53ab73-49f2-42c1-b255-f59f1404a495-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.855492 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.855293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xjg6\" (UniqueName: \"kubernetes.io/projected/dc53ab73-49f2-42c1-b255-f59f1404a495-kube-api-access-5xjg6\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.855492 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.855377    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc53ab73-49f2-42c1-b255-f59f1404a495-tls-certs\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.855492 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.855420    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-home\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.855492 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.855436    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-dshm\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.855689 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.855502    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.855689 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.855604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-model-cache\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.942237 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.942206    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"]
Sep 16 16:28:06.946166 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.946134    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:06.957748 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.957669    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc53ab73-49f2-42c1-b255-f59f1404a495-tls-certs\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.957748 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.957722    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-home\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.957748 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.957746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-dshm\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.957985 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.957795    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.957985 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.957826    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-model-cache\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.957985 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.957881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.957985 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.957906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc53ab73-49f2-42c1-b255-f59f1404a495-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.957985 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.957947    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xjg6\" (UniqueName: \"kubernetes.io/projected/dc53ab73-49f2-42c1-b255-f59f1404a495-kube-api-access-5xjg6\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.958938 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.958659    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.959052 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.958965    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-model-cache\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.959407 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.959363    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.959484 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.959444    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-home\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.960077 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.960053    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc53ab73-49f2-42c1-b255-f59f1404a495-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.960601 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.960577    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"]
Sep 16 16:28:06.961977 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.961937    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc53ab73-49f2-42c1-b255-f59f1404a495-tls-certs\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.962258 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.962233    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-dshm\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:06.971832 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:06.971806    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xjg6\" (UniqueName: \"kubernetes.io/projected/dc53ab73-49f2-42c1-b255-f59f1404a495-kube-api-access-5xjg6\") pod \"tokenizer-template-args-test-kserve-6c7f68f44-p4qbb\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:07.019724 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.019689    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:07.059341 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.059302    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:07.059545 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.059409    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:07.059545 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.059464    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:07.059545 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.059511    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:07.160370 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.160333    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:07.160571 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.160394    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:07.160571 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.160433    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:07.160571 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.160467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:07.160904 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.160880    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:07.160986 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.160936    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:07.161138 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.161112    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:07.162797 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.162781    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:07.260051 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.259964    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:07.295283 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.295258    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"]
Sep 16 16:28:07.297094 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:28:07.297066    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc53ab73_49f2_42c1_b255_f59f1404a495.slice/crio-3c288c79a51eedb024ae660d95be0a4882a9f09fd79fe7a3be2c59481af13518 WatchSource:0}: Error finding container 3c288c79a51eedb024ae660d95be0a4882a9f09fd79fe7a3be2c59481af13518: Status 404 returned error can't find the container with id 3c288c79a51eedb024ae660d95be0a4882a9f09fd79fe7a3be2c59481af13518
Sep 16 16:28:07.534878 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:07.534852    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"]
Sep 16 16:28:07.536911 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:28:07.536887    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9aa81d_08a1_4cb5_97d6_76fe0133ed77.slice/crio-9a6200d04a32488198e6fe5668fd956845d045e264fe99ca28e2bacd81bc11b5 WatchSource:0}: Error finding container 9a6200d04a32488198e6fe5668fd956845d045e264fe99ca28e2bacd81bc11b5: Status 404 returned error can't find the container with id 9a6200d04a32488198e6fe5668fd956845d045e264fe99ca28e2bacd81bc11b5
Sep 16 16:28:08.061178 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:08.061136    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb" event={"ID":"dc53ab73-49f2-42c1-b255-f59f1404a495","Type":"ContainerStarted","Data":"5947d37194207353263c87d0fcd9d1598e210b6a7a9a59b08f5822a547a64038"}
Sep 16 16:28:08.061689 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:08.061207    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb" event={"ID":"dc53ab73-49f2-42c1-b255-f59f1404a495","Type":"ContainerStarted","Data":"3c288c79a51eedb024ae660d95be0a4882a9f09fd79fe7a3be2c59481af13518"}
Sep 16 16:28:08.062862 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:08.062831    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s" event={"ID":"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77","Type":"ContainerStarted","Data":"b299fe2b177ef8f71d10c4730db6da13c1aefb1cb1f96c7621bebca10cb01a5b"}
Sep 16 16:28:08.062987 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:08.062886    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s" event={"ID":"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77","Type":"ContainerStarted","Data":"9a6200d04a32488198e6fe5668fd956845d045e264fe99ca28e2bacd81bc11b5"}
Sep 16 16:28:10.084350 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:10.084308    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc53ab73-49f2-42c1-b255-f59f1404a495" containerID="5947d37194207353263c87d0fcd9d1598e210b6a7a9a59b08f5822a547a64038" exitCode=0
Sep 16 16:28:10.084740 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:10.084384    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb" event={"ID":"dc53ab73-49f2-42c1-b255-f59f1404a495","Type":"ContainerDied","Data":"5947d37194207353263c87d0fcd9d1598e210b6a7a9a59b08f5822a547a64038"}
Sep 16 16:28:11.090919 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:11.090883    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb" event={"ID":"dc53ab73-49f2-42c1-b255-f59f1404a495","Type":"ContainerStarted","Data":"75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785"}
Sep 16 16:28:11.092374 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:11.092352    2574 generic.go:356] "Generic (PLEG): container finished" podID="4a9aa81d-08a1-4cb5-97d6-76fe0133ed77" containerID="b299fe2b177ef8f71d10c4730db6da13c1aefb1cb1f96c7621bebca10cb01a5b" exitCode=0
Sep 16 16:28:11.092478 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:11.092412    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s" event={"ID":"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77","Type":"ContainerDied","Data":"b299fe2b177ef8f71d10c4730db6da13c1aefb1cb1f96c7621bebca10cb01a5b"}
Sep 16 16:28:11.110336 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:11.109853    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb" podStartSLOduration=2.602559169 podStartE2EDuration="5.109834037s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.507274868s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:28:11.109273644 +0000 UTC m=+2287.628896420" watchObservedRunningTime="2026-09-16 16:28:11.109834037 +0000 UTC m=+2287.629456814"
Sep 16 16:28:12.020657 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:12.020625    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:12.098949 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:12.098909    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s" event={"ID":"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77","Type":"ContainerStarted","Data":"bb67fea401e7694cfd06a632f5e634606f7263a6d37cdc71969570d531ffffaa"}
Sep 16 16:28:12.099326 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:12.099253    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:12.116539 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:12.116470    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s" podStartSLOduration=3.632256231 podStartE2EDuration="6.116457506s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.484201275s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:28:12.115711881 +0000 UTC m=+2288.635334657" watchObservedRunningTime="2026-09-16 16:28:12.116457506 +0000 UTC m=+2288.636080281"
Sep 16 16:28:17.020488 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:17.020453    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:17.032997 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:17.032976    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:17.139896 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:17.139863    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:43.114483 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:43.114449    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:44.506707 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:44.506677    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s_4a9aa81d-08a1-4cb5-97d6-76fe0133ed77/main/0.log"
Sep 16 16:28:44.698059 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:44.698020    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"]
Sep 16 16:28:44.710247 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:44.710204    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"]
Sep 16 16:28:44.710456 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:44.710431    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s" podUID="4a9aa81d-08a1-4cb5-97d6-76fe0133ed77" containerName="main" containerID="cri-o://bb67fea401e7694cfd06a632f5e634606f7263a6d37cdc71969570d531ffffaa" gracePeriod=30
Sep 16 16:28:44.711469 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:28:44.711350    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:28:44.711469 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:28:44.711469 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:28:44.711469 ip-10-0-143-143 kubenswrapper[2574]:  > containerID="75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785" cmd=["/bin/sleep","15"]
Sep 16 16:28:44.711469 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:44.711406    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb" message="" err=<
Sep 16 16:28:44.711469 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:28:44.711469 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:28:44.711469 ip-10-0-143-143 kubenswrapper[2574]:  >
Sep 16 16:28:44.711469 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:28:44.711423    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:28:44.711469 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:28:44.711469 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:28:44.711469 ip-10-0-143-143 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb" podUID="dc53ab73-49f2-42c1-b255-f59f1404a495" containerName="main" containerID="cri-o://75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785"
Sep 16 16:28:44.711974 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:44.711478    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb" podUID="dc53ab73-49f2-42c1-b255-f59f1404a495" containerName="main" containerID="cri-o://75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785" gracePeriod=60
Sep 16 16:28:45.069708 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.069650    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:45.218277 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.217248    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-home\") pod \"dc53ab73-49f2-42c1-b255-f59f1404a495\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") "
Sep 16 16:28:45.218277 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.217308    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-dshm\") pod \"dc53ab73-49f2-42c1-b255-f59f1404a495\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") "
Sep 16 16:28:45.218277 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.217338    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc53ab73-49f2-42c1-b255-f59f1404a495-cabundle-cert\") pod \"dc53ab73-49f2-42c1-b255-f59f1404a495\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") "
Sep 16 16:28:45.218277 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.217396    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xjg6\" (UniqueName: \"kubernetes.io/projected/dc53ab73-49f2-42c1-b255-f59f1404a495-kube-api-access-5xjg6\") pod \"dc53ab73-49f2-42c1-b255-f59f1404a495\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") "
Sep 16 16:28:45.218277 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.217441    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-tmp-dir\") pod \"dc53ab73-49f2-42c1-b255-f59f1404a495\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") "
Sep 16 16:28:45.218277 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.217468    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-model-cache\") pod \"dc53ab73-49f2-42c1-b255-f59f1404a495\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") "
Sep 16 16:28:45.218277 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.217512    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-kserve-provision-location\") pod \"dc53ab73-49f2-42c1-b255-f59f1404a495\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") "
Sep 16 16:28:45.218277 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.217578    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc53ab73-49f2-42c1-b255-f59f1404a495-tls-certs\") pod \"dc53ab73-49f2-42c1-b255-f59f1404a495\" (UID: \"dc53ab73-49f2-42c1-b255-f59f1404a495\") "
Sep 16 16:28:45.220416 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.220073    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-home" (OuterVolumeSpecName: "home") pod "dc53ab73-49f2-42c1-b255-f59f1404a495" (UID: "dc53ab73-49f2-42c1-b255-f59f1404a495"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:28:45.220416 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.220298    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc53ab73-49f2-42c1-b255-f59f1404a495" (UID: "dc53ab73-49f2-42c1-b255-f59f1404a495"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:28:45.220604 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.220436    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc53ab73-49f2-42c1-b255-f59f1404a495" (UID: "dc53ab73-49f2-42c1-b255-f59f1404a495"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:28:45.220853 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.220819    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc53ab73-49f2-42c1-b255-f59f1404a495-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc53ab73-49f2-42c1-b255-f59f1404a495" (UID: "dc53ab73-49f2-42c1-b255-f59f1404a495"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:28:45.234157 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.234113    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc53ab73-49f2-42c1-b255-f59f1404a495-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc53ab73-49f2-42c1-b255-f59f1404a495" (UID: "dc53ab73-49f2-42c1-b255-f59f1404a495"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:28:45.234157 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.234142    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc53ab73-49f2-42c1-b255-f59f1404a495-kube-api-access-5xjg6" (OuterVolumeSpecName: "kube-api-access-5xjg6") pod "dc53ab73-49f2-42c1-b255-f59f1404a495" (UID: "dc53ab73-49f2-42c1-b255-f59f1404a495"). InnerVolumeSpecName "kube-api-access-5xjg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:28:45.236254 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.236228    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-dshm" (OuterVolumeSpecName: "dshm") pod "dc53ab73-49f2-42c1-b255-f59f1404a495" (UID: "dc53ab73-49f2-42c1-b255-f59f1404a495"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:28:45.249755 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.249728    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc53ab73-49f2-42c1-b255-f59f1404a495" (UID: "dc53ab73-49f2-42c1-b255-f59f1404a495"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:28:45.272384 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.272352    2574 generic.go:356] "Generic (PLEG): container finished" podID="4a9aa81d-08a1-4cb5-97d6-76fe0133ed77" containerID="bb67fea401e7694cfd06a632f5e634606f7263a6d37cdc71969570d531ffffaa" exitCode=0
Sep 16 16:28:45.272494 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.272425    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s" event={"ID":"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77","Type":"ContainerDied","Data":"bb67fea401e7694cfd06a632f5e634606f7263a6d37cdc71969570d531ffffaa"}
Sep 16 16:28:45.272494 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.272471    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s" event={"ID":"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77","Type":"ContainerDied","Data":"9a6200d04a32488198e6fe5668fd956845d045e264fe99ca28e2bacd81bc11b5"}
Sep 16 16:28:45.272627 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.272492    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a6200d04a32488198e6fe5668fd956845d045e264fe99ca28e2bacd81bc11b5"
Sep 16 16:28:45.272670 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.272649    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:45.277026 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.277005    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc53ab73-49f2-42c1-b255-f59f1404a495" containerID="75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785" exitCode=0
Sep 16 16:28:45.277136 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.277055    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"
Sep 16 16:28:45.277136 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.277058    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb" event={"ID":"dc53ab73-49f2-42c1-b255-f59f1404a495","Type":"ContainerDied","Data":"75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785"}
Sep 16 16:28:45.277245 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.277168    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb" event={"ID":"dc53ab73-49f2-42c1-b255-f59f1404a495","Type":"ContainerDied","Data":"3c288c79a51eedb024ae660d95be0a4882a9f09fd79fe7a3be2c59481af13518"}
Sep 16 16:28:45.277245 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.277190    2574 scope.go:117] "RemoveContainer" containerID="75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785"
Sep 16 16:28:45.286032 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.286014    2574 scope.go:117] "RemoveContainer" containerID="5947d37194207353263c87d0fcd9d1598e210b6a7a9a59b08f5822a547a64038"
Sep 16 16:28:45.295352 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.295329    2574 scope.go:117] "RemoveContainer" containerID="75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785"
Sep 16 16:28:45.295658 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:28:45.295634    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785\": container with ID starting with 75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785 not found: ID does not exist" containerID="75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785"
Sep 16 16:28:45.295728 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.295665    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785"} err="failed to get container status \"75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785\": rpc error: code = NotFound desc = could not find container \"75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785\": container with ID starting with 75c332f717b266699db474520e04f4ccef2ba8aa105082ad4122129250c94785 not found: ID does not exist"
Sep 16 16:28:45.295728 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.295684    2574 scope.go:117] "RemoveContainer" containerID="5947d37194207353263c87d0fcd9d1598e210b6a7a9a59b08f5822a547a64038"
Sep 16 16:28:45.295963 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:28:45.295939    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5947d37194207353263c87d0fcd9d1598e210b6a7a9a59b08f5822a547a64038\": container with ID starting with 5947d37194207353263c87d0fcd9d1598e210b6a7a9a59b08f5822a547a64038 not found: ID does not exist" containerID="5947d37194207353263c87d0fcd9d1598e210b6a7a9a59b08f5822a547a64038"
Sep 16 16:28:45.296008 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.295973    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5947d37194207353263c87d0fcd9d1598e210b6a7a9a59b08f5822a547a64038"} err="failed to get container status \"5947d37194207353263c87d0fcd9d1598e210b6a7a9a59b08f5822a547a64038\": rpc error: code = NotFound desc = could not find container \"5947d37194207353263c87d0fcd9d1598e210b6a7a9a59b08f5822a547a64038\": container with ID starting with 5947d37194207353263c87d0fcd9d1598e210b6a7a9a59b08f5822a547a64038 not found: ID does not exist"
Sep 16 16:28:45.303353 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.303323    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"]
Sep 16 16:28:45.307927 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.307906    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6c7f68f44-p4qbb"]
Sep 16 16:28:45.318501 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.318475    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.318501 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.318498    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc53ab73-49f2-42c1-b255-f59f1404a495-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.318659 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.318511    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xjg6\" (UniqueName: \"kubernetes.io/projected/dc53ab73-49f2-42c1-b255-f59f1404a495-kube-api-access-5xjg6\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.318659 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.318538    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.318659 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.318551    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.318659 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.318563    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.318659 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.318574    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc53ab73-49f2-42c1-b255-f59f1404a495-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.318659 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.318585    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc53ab73-49f2-42c1-b255-f59f1404a495-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.419377 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.419346    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-kserve-provision-location\") pod \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") "
Sep 16 16:28:45.419546 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.419418    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-tmp-dir\") pod \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") "
Sep 16 16:28:45.419546 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.419474    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-tls-certs\") pod \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") "
Sep 16 16:28:45.419546 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.419509    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-cabundle-cert\") pod \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\" (UID: \"4a9aa81d-08a1-4cb5-97d6-76fe0133ed77\") "
Sep 16 16:28:45.419833 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.419811    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a9aa81d-08a1-4cb5-97d6-76fe0133ed77" (UID: "4a9aa81d-08a1-4cb5-97d6-76fe0133ed77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:28:45.419947 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.419927    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a9aa81d-08a1-4cb5-97d6-76fe0133ed77" (UID: "4a9aa81d-08a1-4cb5-97d6-76fe0133ed77"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:28:45.421463 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.421445    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a9aa81d-08a1-4cb5-97d6-76fe0133ed77" (UID: "4a9aa81d-08a1-4cb5-97d6-76fe0133ed77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:28:45.441205 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.441179    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a9aa81d-08a1-4cb5-97d6-76fe0133ed77" (UID: "4a9aa81d-08a1-4cb5-97d6-76fe0133ed77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:28:45.521145 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.521119    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.521145 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.521143    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.521145 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.521152    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.521635 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.521161    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:28:45.994639 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:45.994609    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc53ab73-49f2-42c1-b255-f59f1404a495" path="/var/lib/kubelet/pods/dc53ab73-49f2-42c1-b255-f59f1404a495/volumes"
Sep 16 16:28:46.282719 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:46.282683    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"
Sep 16 16:28:46.298586 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:46.298558    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"]
Sep 16 16:28:46.304841 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:46.304817    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65d76d4d5d-54c8s"]
Sep 16 16:28:47.994645 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:47.994616    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a9aa81d-08a1-4cb5-97d6-76fe0133ed77" path="/var/lib/kubelet/pods/4a9aa81d-08a1-4cb5-97d6-76fe0133ed77/volumes"
Sep 16 16:28:50.587975 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:50.587940    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"]
Sep 16 16:28:59.963742 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.963662    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"]
Sep 16 16:28:59.964168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.964110    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc53ab73-49f2-42c1-b255-f59f1404a495" containerName="storage-initializer"
Sep 16 16:28:59.964168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.964124    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc53ab73-49f2-42c1-b255-f59f1404a495" containerName="storage-initializer"
Sep 16 16:28:59.964168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.964137    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9aa81d-08a1-4cb5-97d6-76fe0133ed77" containerName="main"
Sep 16 16:28:59.964168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.964143    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9aa81d-08a1-4cb5-97d6-76fe0133ed77" containerName="main"
Sep 16 16:28:59.964168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.964150    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9aa81d-08a1-4cb5-97d6-76fe0133ed77" containerName="storage-initializer"
Sep 16 16:28:59.964168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.964156    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9aa81d-08a1-4cb5-97d6-76fe0133ed77" containerName="storage-initializer"
Sep 16 16:28:59.964168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.964162    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc53ab73-49f2-42c1-b255-f59f1404a495" containerName="main"
Sep 16 16:28:59.964168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.964167    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc53ab73-49f2-42c1-b255-f59f1404a495" containerName="main"
Sep 16 16:28:59.964556 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.964239    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a9aa81d-08a1-4cb5-97d6-76fe0133ed77" containerName="main"
Sep 16 16:28:59.964556 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.964246    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc53ab73-49f2-42c1-b255-f59f1404a495" containerName="main"
Sep 16 16:28:59.969214 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.969192    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:28:59.972760 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.972737    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 16 16:28:59.972886 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.972777    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 16 16:28:59.972886 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.972793    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-4ksct\""
Sep 16 16:28:59.972886 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.972858    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 16 16:28:59.977091 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:28:59.977070    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"]
Sep 16 16:29:00.053468 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.053435    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-dshm\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.053468 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.053468    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d85e52d-c02e-4135-8e56-d56991ae73dc-tls-certs\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.053695 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.053486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.053695 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.053599    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6xrh\" (UniqueName: \"kubernetes.io/projected/8d85e52d-c02e-4135-8e56-d56991ae73dc-kube-api-access-l6xrh\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.053695 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.053640    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-home\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.053695 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.053686    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-model-cache\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.154330 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.154292    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-model-cache\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.154552 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.154370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-dshm\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.154552 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.154394    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d85e52d-c02e-4135-8e56-d56991ae73dc-tls-certs\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.154552 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.154423    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.154731 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.154616    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6xrh\" (UniqueName: \"kubernetes.io/projected/8d85e52d-c02e-4135-8e56-d56991ae73dc-kube-api-access-l6xrh\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.154731 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.154692    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-home\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.154847 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.154751    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-model-cache\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.154847 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.154808    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.154978 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.154958    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-home\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.156746 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.156718    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-dshm\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.156871 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.156856    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d85e52d-c02e-4135-8e56-d56991ae73dc-tls-certs\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.165091 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.165063    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6xrh\" (UniqueName: \"kubernetes.io/projected/8d85e52d-c02e-4135-8e56-d56991ae73dc-kube-api-access-l6xrh\") pod \"leave-v1-kserve-65684d4f5-x745m\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.281829 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.281744    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:00.555281 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:00.555197    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"]
Sep 16 16:29:00.559253 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:29:00.559225    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d85e52d_c02e_4135_8e56_d56991ae73dc.slice/crio-713999e451d89629f6753b3edf7571401410e9e6da6e588cdbb0c6219c67abb6 WatchSource:0}: Error finding container 713999e451d89629f6753b3edf7571401410e9e6da6e588cdbb0c6219c67abb6: Status 404 returned error can't find the container with id 713999e451d89629f6753b3edf7571401410e9e6da6e588cdbb0c6219c67abb6
Sep 16 16:29:01.359052 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:01.359012    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m" event={"ID":"8d85e52d-c02e-4135-8e56-d56991ae73dc","Type":"ContainerStarted","Data":"f1d63fd1471379bc338b52d371e7dc3d0f31bc2aba135feb421ce9f5aa89c45c"}
Sep 16 16:29:01.359432 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:01.359076    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m" event={"ID":"8d85e52d-c02e-4135-8e56-d56991ae73dc","Type":"ContainerStarted","Data":"713999e451d89629f6753b3edf7571401410e9e6da6e588cdbb0c6219c67abb6"}
Sep 16 16:29:01.379840 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:01.379785    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m" podStartSLOduration=2.379770161 podStartE2EDuration="2.379770161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:29:01.378308349 +0000 UTC m=+2337.897931125" watchObservedRunningTime="2026-09-16 16:29:01.379770161 +0000 UTC m=+2337.899392937"
Sep 16 16:29:02.282077 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:02.282043    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:05.606227 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.606185    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf" podUID="a2cf8e6a-0e6f-498b-8e56-224469d47b5d" containerName="main" containerID="cri-o://670c26a6ca180ebd9b0cce823e7934a53b7512533ce94016c5bf6d20d2bc2604" gracePeriod=45
Sep 16 16:29:05.858600 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.858543    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:29:05.904192 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.904167    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-model-cache\") pod \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") "
Sep 16 16:29:05.904322 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.904199    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-tmp-dir\") pod \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") "
Sep 16 16:29:05.904322 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.904245    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7bw7\" (UniqueName: \"kubernetes.io/projected/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-kube-api-access-x7bw7\") pod \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") "
Sep 16 16:29:05.904322 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.904268    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-tls-certs\") pod \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") "
Sep 16 16:29:05.904458 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.904344    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-dshm\") pod \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") "
Sep 16 16:29:05.904458 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.904386    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-home\") pod \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\" (UID: \"a2cf8e6a-0e6f-498b-8e56-224469d47b5d\") "
Sep 16 16:29:05.904582 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.904448    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2cf8e6a-0e6f-498b-8e56-224469d47b5d" (UID: "a2cf8e6a-0e6f-498b-8e56-224469d47b5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:29:05.904582 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.904463    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2cf8e6a-0e6f-498b-8e56-224469d47b5d" (UID: "a2cf8e6a-0e6f-498b-8e56-224469d47b5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:29:05.904722 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.904698    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:29:05.904722 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.904723    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:29:05.904851 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.904748    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-home" (OuterVolumeSpecName: "home") pod "a2cf8e6a-0e6f-498b-8e56-224469d47b5d" (UID: "a2cf8e6a-0e6f-498b-8e56-224469d47b5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:29:05.906576 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.906547    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-dshm" (OuterVolumeSpecName: "dshm") pod "a2cf8e6a-0e6f-498b-8e56-224469d47b5d" (UID: "a2cf8e6a-0e6f-498b-8e56-224469d47b5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:29:05.906666 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.906634    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2cf8e6a-0e6f-498b-8e56-224469d47b5d" (UID: "a2cf8e6a-0e6f-498b-8e56-224469d47b5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:29:05.906666 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:05.906642    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-kube-api-access-x7bw7" (OuterVolumeSpecName: "kube-api-access-x7bw7") pod "a2cf8e6a-0e6f-498b-8e56-224469d47b5d" (UID: "a2cf8e6a-0e6f-498b-8e56-224469d47b5d"). InnerVolumeSpecName "kube-api-access-x7bw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:29:06.005818 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:06.005792    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7bw7\" (UniqueName: \"kubernetes.io/projected/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-kube-api-access-x7bw7\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:29:06.005818 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:06.005818    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:29:06.005973 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:06.005830    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:29:06.005973 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:06.005842    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2cf8e6a-0e6f-498b-8e56-224469d47b5d-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:29:06.384195 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:06.384164    2574 generic.go:356] "Generic (PLEG): container finished" podID="a2cf8e6a-0e6f-498b-8e56-224469d47b5d" containerID="670c26a6ca180ebd9b0cce823e7934a53b7512533ce94016c5bf6d20d2bc2604" exitCode=0
Sep 16 16:29:06.384359 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:06.384202    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"
Sep 16 16:29:06.384359 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:06.384206    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf" event={"ID":"a2cf8e6a-0e6f-498b-8e56-224469d47b5d","Type":"ContainerDied","Data":"670c26a6ca180ebd9b0cce823e7934a53b7512533ce94016c5bf6d20d2bc2604"}
Sep 16 16:29:06.384359 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:06.384247    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf" event={"ID":"a2cf8e6a-0e6f-498b-8e56-224469d47b5d","Type":"ContainerDied","Data":"cdca3b66e78e4bb81cf4dc76ca92e50f42c5f774410114a25e6e3dcc88b62b31"}
Sep 16 16:29:06.384359 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:06.384267    2574 scope.go:117] "RemoveContainer" containerID="670c26a6ca180ebd9b0cce823e7934a53b7512533ce94016c5bf6d20d2bc2604"
Sep 16 16:29:06.392639 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:06.392622    2574 scope.go:117] "RemoveContainer" containerID="670c26a6ca180ebd9b0cce823e7934a53b7512533ce94016c5bf6d20d2bc2604"
Sep 16 16:29:06.392903 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:29:06.392886    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670c26a6ca180ebd9b0cce823e7934a53b7512533ce94016c5bf6d20d2bc2604\": container with ID starting with 670c26a6ca180ebd9b0cce823e7934a53b7512533ce94016c5bf6d20d2bc2604 not found: ID does not exist" containerID="670c26a6ca180ebd9b0cce823e7934a53b7512533ce94016c5bf6d20d2bc2604"
Sep 16 16:29:06.392969 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:06.392914    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670c26a6ca180ebd9b0cce823e7934a53b7512533ce94016c5bf6d20d2bc2604"} err="failed to get container status \"670c26a6ca180ebd9b0cce823e7934a53b7512533ce94016c5bf6d20d2bc2604\": rpc error: code = NotFound desc = could not find container \"670c26a6ca180ebd9b0cce823e7934a53b7512533ce94016c5bf6d20d2bc2604\": container with ID starting with 670c26a6ca180ebd9b0cce823e7934a53b7512533ce94016c5bf6d20d2bc2604 not found: ID does not exist"
Sep 16 16:29:06.401444 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:06.401422    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"]
Sep 16 16:29:06.407437 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:06.407417    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-b6jmf"]
Sep 16 16:29:07.994285 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:07.994254    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2cf8e6a-0e6f-498b-8e56-224469d47b5d" path="/var/lib/kubelet/pods/a2cf8e6a-0e6f-498b-8e56-224469d47b5d/volumes"
Sep 16 16:29:10.282362 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:10.282317    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:10.294555 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:10.294530    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:29:10.416403 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:29:10.416374    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:30:42.484268 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:42.484230    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"]
Sep 16 16:30:57.510239 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.510175    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m" podUID="8d85e52d-c02e-4135-8e56-d56991ae73dc" containerName="main" containerID="cri-o://f1d63fd1471379bc338b52d371e7dc3d0f31bc2aba135feb421ce9f5aa89c45c" gracePeriod=45
Sep 16 16:30:57.755305 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.755282    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:30:57.894102 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.894068    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-model-cache\") pod \"8d85e52d-c02e-4135-8e56-d56991ae73dc\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") "
Sep 16 16:30:57.894253 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.894128    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6xrh\" (UniqueName: \"kubernetes.io/projected/8d85e52d-c02e-4135-8e56-d56991ae73dc-kube-api-access-l6xrh\") pod \"8d85e52d-c02e-4135-8e56-d56991ae73dc\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") "
Sep 16 16:30:57.894253 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.894167    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d85e52d-c02e-4135-8e56-d56991ae73dc-tls-certs\") pod \"8d85e52d-c02e-4135-8e56-d56991ae73dc\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") "
Sep 16 16:30:57.894253 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.894196    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-tmp-dir\") pod \"8d85e52d-c02e-4135-8e56-d56991ae73dc\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") "
Sep 16 16:30:57.894253 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.894211    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-home\") pod \"8d85e52d-c02e-4135-8e56-d56991ae73dc\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") "
Sep 16 16:30:57.894558 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.894279    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-dshm\") pod \"8d85e52d-c02e-4135-8e56-d56991ae73dc\" (UID: \"8d85e52d-c02e-4135-8e56-d56991ae73dc\") "
Sep 16 16:30:57.894558 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.894339    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d85e52d-c02e-4135-8e56-d56991ae73dc" (UID: "8d85e52d-c02e-4135-8e56-d56991ae73dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:30:57.894558 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.894495    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d85e52d-c02e-4135-8e56-d56991ae73dc" (UID: "8d85e52d-c02e-4135-8e56-d56991ae73dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:30:57.894558 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.894550    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-home" (OuterVolumeSpecName: "home") pod "8d85e52d-c02e-4135-8e56-d56991ae73dc" (UID: "8d85e52d-c02e-4135-8e56-d56991ae73dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:30:57.894714 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.894621    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:30:57.894714 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.894635    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:30:57.894714 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.894644    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:30:57.896371 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.896348    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-dshm" (OuterVolumeSpecName: "dshm") pod "8d85e52d-c02e-4135-8e56-d56991ae73dc" (UID: "8d85e52d-c02e-4135-8e56-d56991ae73dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:30:57.896722 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.896701    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d85e52d-c02e-4135-8e56-d56991ae73dc-kube-api-access-l6xrh" (OuterVolumeSpecName: "kube-api-access-l6xrh") pod "8d85e52d-c02e-4135-8e56-d56991ae73dc" (UID: "8d85e52d-c02e-4135-8e56-d56991ae73dc"). InnerVolumeSpecName "kube-api-access-l6xrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:30:57.896722 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.896714    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d85e52d-c02e-4135-8e56-d56991ae73dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d85e52d-c02e-4135-8e56-d56991ae73dc" (UID: "8d85e52d-c02e-4135-8e56-d56991ae73dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:30:57.914046 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.914021    2574 generic.go:356] "Generic (PLEG): container finished" podID="8d85e52d-c02e-4135-8e56-d56991ae73dc" containerID="f1d63fd1471379bc338b52d371e7dc3d0f31bc2aba135feb421ce9f5aa89c45c" exitCode=0
Sep 16 16:30:57.914157 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.914062    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"
Sep 16 16:30:57.914157 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.914078    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m" event={"ID":"8d85e52d-c02e-4135-8e56-d56991ae73dc","Type":"ContainerDied","Data":"f1d63fd1471379bc338b52d371e7dc3d0f31bc2aba135feb421ce9f5aa89c45c"}
Sep 16 16:30:57.914157 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.914127    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m" event={"ID":"8d85e52d-c02e-4135-8e56-d56991ae73dc","Type":"ContainerDied","Data":"713999e451d89629f6753b3edf7571401410e9e6da6e588cdbb0c6219c67abb6"}
Sep 16 16:30:57.914157 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.914155    2574 scope.go:117] "RemoveContainer" containerID="f1d63fd1471379bc338b52d371e7dc3d0f31bc2aba135feb421ce9f5aa89c45c"
Sep 16 16:30:57.923241 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.923223    2574 scope.go:117] "RemoveContainer" containerID="f1d63fd1471379bc338b52d371e7dc3d0f31bc2aba135feb421ce9f5aa89c45c"
Sep 16 16:30:57.923494 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:30:57.923477    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d63fd1471379bc338b52d371e7dc3d0f31bc2aba135feb421ce9f5aa89c45c\": container with ID starting with f1d63fd1471379bc338b52d371e7dc3d0f31bc2aba135feb421ce9f5aa89c45c not found: ID does not exist" containerID="f1d63fd1471379bc338b52d371e7dc3d0f31bc2aba135feb421ce9f5aa89c45c"
Sep 16 16:30:57.923590 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.923504    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d63fd1471379bc338b52d371e7dc3d0f31bc2aba135feb421ce9f5aa89c45c"} err="failed to get container status \"f1d63fd1471379bc338b52d371e7dc3d0f31bc2aba135feb421ce9f5aa89c45c\": rpc error: code = NotFound desc = could not find container \"f1d63fd1471379bc338b52d371e7dc3d0f31bc2aba135feb421ce9f5aa89c45c\": container with ID starting with f1d63fd1471379bc338b52d371e7dc3d0f31bc2aba135feb421ce9f5aa89c45c not found: ID does not exist"
Sep 16 16:30:57.934968 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.934945    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"]
Sep 16 16:30:57.938412 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.938389    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-x745m"]
Sep 16 16:30:57.994050 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.994026    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d85e52d-c02e-4135-8e56-d56991ae73dc" path="/var/lib/kubelet/pods/8d85e52d-c02e-4135-8e56-d56991ae73dc/volumes"
Sep 16 16:30:57.995278 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.995263    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d85e52d-c02e-4135-8e56-d56991ae73dc-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:30:57.995331 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.995280    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d85e52d-c02e-4135-8e56-d56991ae73dc-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:30:57.995331 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:30:57.995289    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6xrh\" (UniqueName: \"kubernetes.io/projected/8d85e52d-c02e-4135-8e56-d56991ae73dc-kube-api-access-l6xrh\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:31:09.873174 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.873142    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"]
Sep 16 16:31:09.873593 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.873577    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2cf8e6a-0e6f-498b-8e56-224469d47b5d" containerName="main"
Sep 16 16:31:09.873647 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.873596    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2cf8e6a-0e6f-498b-8e56-224469d47b5d" containerName="main"
Sep 16 16:31:09.873647 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.873628    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d85e52d-c02e-4135-8e56-d56991ae73dc" containerName="main"
Sep 16 16:31:09.873647 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.873633    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d85e52d-c02e-4135-8e56-d56991ae73dc" containerName="main"
Sep 16 16:31:09.873756 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.873688    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2cf8e6a-0e6f-498b-8e56-224469d47b5d" containerName="main"
Sep 16 16:31:09.873756 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.873697    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d85e52d-c02e-4135-8e56-d56991ae73dc" containerName="main"
Sep 16 16:31:09.878506 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.878487    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:09.881900 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.881880    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 16 16:31:09.882014 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.881907    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 16 16:31:09.882014 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.881877    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-rvvz6\""
Sep 16 16:31:09.882014 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.881886    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 16 16:31:09.886611 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.886587    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"]
Sep 16 16:31:09.995403 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.995369    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-home\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:09.995403 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.995398    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk5gk\" (UniqueName: \"kubernetes.io/projected/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-kube-api-access-pk5gk\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:09.995688 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.995418    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:09.995688 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.995514    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:09.995688 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.995583    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-dshm\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:09.995688 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:09.995616    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:10.096387 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.096351    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:10.096591 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.096402    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-dshm\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:10.096591 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.096429    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:10.096591 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.096540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-home\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:10.096591 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.096565    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk5gk\" (UniqueName: \"kubernetes.io/projected/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-kube-api-access-pk5gk\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:10.096819 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.096592    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:10.096876 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.096815    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:10.096936 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.096885    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-home\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:10.096983 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.096933    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:10.098702 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.098677    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-dshm\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:10.098947 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.098930    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:10.103973 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.103951    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk5gk\" (UniqueName: \"kubernetes.io/projected/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-kube-api-access-pk5gk\") pod \"tri-v1-kserve-79f8f98bc5-rdzz8\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:10.188438 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.188378    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:10.459377 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.459297    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"]
Sep 16 16:31:10.462507 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:31:10.462479    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca92abe9_6d0c_43f5_8061_4d0fd16255e5.slice/crio-3549093882cb9c31cacb4c851ddb38b49d739d85ec19e0314336faed5a80c4b4 WatchSource:0}: Error finding container 3549093882cb9c31cacb4c851ddb38b49d739d85ec19e0314336faed5a80c4b4: Status 404 returned error can't find the container with id 3549093882cb9c31cacb4c851ddb38b49d739d85ec19e0314336faed5a80c4b4
Sep 16 16:31:10.980281 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.980243    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8" event={"ID":"ca92abe9-6d0c-43f5-8061-4d0fd16255e5","Type":"ContainerStarted","Data":"bdc908ae7c4d8b02987ef5532bd39228a27cefe1028c0b74988484903246da16"}
Sep 16 16:31:10.980787 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:10.980303    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8" event={"ID":"ca92abe9-6d0c-43f5-8061-4d0fd16255e5","Type":"ContainerStarted","Data":"3549093882cb9c31cacb4c851ddb38b49d739d85ec19e0314336faed5a80c4b4"}
Sep 16 16:31:11.018210 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:11.018157    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8" podStartSLOduration=2.018140404 podStartE2EDuration="2.018140404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:31:11.015420232 +0000 UTC m=+2467.535043007" watchObservedRunningTime="2026-09-16 16:31:11.018140404 +0000 UTC m=+2467.537763179"
Sep 16 16:31:11.188494 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:11.188459    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:20.189490 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:20.189456    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:20.201576 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:20.201551    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:21.037924 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:21.037891    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:31:23.172074 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.172041    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"]
Sep 16 16:31:23.175494 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.175476    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.177757 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.177737    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 16 16:31:23.185552 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.185508    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"]
Sep 16 16:31:23.205753 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.205724    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-dshm\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.205846 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.205755    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.205846 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.205785    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k7p7\" (UniqueName: \"kubernetes.io/projected/4349f77a-20eb-4db3-abdd-512b5c77135c-kube-api-access-8k7p7\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.205924 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.205855    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-model-cache\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.205924 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.205914    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4349f77a-20eb-4db3-abdd-512b5c77135c-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.206018 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.205993    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-home\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.306688 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.306660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-dshm\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.306688 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.306690    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.306882 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.306721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k7p7\" (UniqueName: \"kubernetes.io/projected/4349f77a-20eb-4db3-abdd-512b5c77135c-kube-api-access-8k7p7\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.306882 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.306753    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-model-cache\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.306882 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.306820    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4349f77a-20eb-4db3-abdd-512b5c77135c-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.306882 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.306858    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-home\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.307185 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.307162    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.307376 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.307351    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-home\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.307376 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.307366    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-model-cache\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.308996 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.308967    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-dshm\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.309804 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.309780    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4349f77a-20eb-4db3-abdd-512b5c77135c-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.314085 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.314066    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k7p7\" (UniqueName: \"kubernetes.io/projected/4349f77a-20eb-4db3-abdd-512b5c77135c-kube-api-access-8k7p7\") pod \"tri-v2-kserve-dcb554bf5-rk26s\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.487380 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.487296    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:23.759331 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:23.759304    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"]
Sep 16 16:31:23.761056 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:31:23.761023    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4349f77a_20eb_4db3_abdd_512b5c77135c.slice/crio-2569bbf687bca9a12d29f34b4db68c426962f79ad238251d0b5e647f8958a0e1 WatchSource:0}: Error finding container 2569bbf687bca9a12d29f34b4db68c426962f79ad238251d0b5e647f8958a0e1: Status 404 returned error can't find the container with id 2569bbf687bca9a12d29f34b4db68c426962f79ad238251d0b5e647f8958a0e1
Sep 16 16:31:24.043960 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:24.043865    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s" event={"ID":"4349f77a-20eb-4db3-abdd-512b5c77135c","Type":"ContainerStarted","Data":"89a1ac7d12f047473238c0855e1d9c439475fcdf75962b1510778f4901f653e5"}
Sep 16 16:31:24.043960 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:24.043928    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s" event={"ID":"4349f77a-20eb-4db3-abdd-512b5c77135c","Type":"ContainerStarted","Data":"2569bbf687bca9a12d29f34b4db68c426962f79ad238251d0b5e647f8958a0e1"}
Sep 16 16:31:24.064290 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:24.064233    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s" podStartSLOduration=1.064218506 podStartE2EDuration="1.064218506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:31:24.063240583 +0000 UTC m=+2480.582863360" watchObservedRunningTime="2026-09-16 16:31:24.064218506 +0000 UTC m=+2480.583841282"
Sep 16 16:31:24.488298 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:24.488270    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:32.556061 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.556026    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"]
Sep 16 16:31:32.559981 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.559958    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.562122 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.562095    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 16 16:31:32.569104 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.569081    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"]
Sep 16 16:31:32.591632 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.591588    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6sxj\" (UniqueName: \"kubernetes.io/projected/4463d7cf-de55-4c56-9fce-f255a51d718c-kube-api-access-k6sxj\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.591842 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.591723    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.591842 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.591754    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-dshm\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.591842 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.591779    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-home\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.591842 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.591794    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4463d7cf-de55-4c56-9fce-f255a51d718c-tls-certs\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.592071 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.591878    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-model-cache\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.692669 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.692631    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4463d7cf-de55-4c56-9fce-f255a51d718c-tls-certs\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.692862 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.692685    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-model-cache\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.692862 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.692729    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6sxj\" (UniqueName: \"kubernetes.io/projected/4463d7cf-de55-4c56-9fce-f255a51d718c-kube-api-access-k6sxj\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.692862 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.692788    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.692862 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.692806    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-dshm\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.692862 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.692828    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-home\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.693181 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.693158    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-model-cache\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.693253 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.693231    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.693360 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.693343    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-home\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.694941 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.694915    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-dshm\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.695201 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.695183    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4463d7cf-de55-4c56-9fce-f255a51d718c-tls-certs\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.700661 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.700643    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6sxj\" (UniqueName: \"kubernetes.io/projected/4463d7cf-de55-4c56-9fce-f255a51d718c-kube-api-access-k6sxj\") pod \"tri-v3-kserve-798c88c66c-krq69\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:32.872267 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:32.872234    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:33.143196 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:33.143169    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"]
Sep 16 16:31:33.145343 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:31:33.145316    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4463d7cf_de55_4c56_9fce_f255a51d718c.slice/crio-153122c9bbf5835e11f01621691566cad4a1c2a095c4307e2a93829de28d559c WatchSource:0}: Error finding container 153122c9bbf5835e11f01621691566cad4a1c2a095c4307e2a93829de28d559c: Status 404 returned error can't find the container with id 153122c9bbf5835e11f01621691566cad4a1c2a095c4307e2a93829de28d559c
Sep 16 16:31:33.488100 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:33.488014    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:33.499964 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:33.499935    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:34.092892 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:34.092850    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69" event={"ID":"4463d7cf-de55-4c56-9fce-f255a51d718c","Type":"ContainerStarted","Data":"999cd451332fac1ddac5858247ba2c5d2f04436a1d05b93fa31e5a3ce8df06f7"}
Sep 16 16:31:34.093278 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:34.092917    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69" event={"ID":"4463d7cf-de55-4c56-9fce-f255a51d718c","Type":"ContainerStarted","Data":"153122c9bbf5835e11f01621691566cad4a1c2a095c4307e2a93829de28d559c"}
Sep 16 16:31:34.103993 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:34.103967    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:31:34.109751 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:34.109706    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69" podStartSLOduration=2.109692539 podStartE2EDuration="2.109692539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:31:34.108056229 +0000 UTC m=+2490.627679006" watchObservedRunningTime="2026-09-16 16:31:34.109692539 +0000 UTC m=+2490.629315313"
Sep 16 16:31:34.872918 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:34.872883    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:42.872416 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:42.872370    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:42.885049 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:42.885023    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:31:43.147376 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:31:43.147292    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:33:38.570085 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.570009    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"]
Sep 16 16:33:38.573794 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.573776    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.576770 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.576721    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 16:33:38.576905 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.576800    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 16:33:38.576905 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.576721    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-ph4nq\""
Sep 16 16:33:38.576905 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.576721    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 16:33:38.584588 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.584566    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"]
Sep 16 16:33:38.653882 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.653840    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.654057 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.653902    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.654057 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.653965    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858b4699-5ef5-46ca-bc2f-b80f700350e7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.654057 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.653998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.654057 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.654030    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7qth\" (UniqueName: \"kubernetes.io/projected/858b4699-5ef5-46ca-bc2f-b80f700350e7-kube-api-access-r7qth\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.654057 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.654047    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.754595 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.754550    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.754761 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.754673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.754761 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.754738    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.754841 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.754775    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858b4699-5ef5-46ca-bc2f-b80f700350e7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.754841 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.754803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.754956 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.754928    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7qth\" (UniqueName: \"kubernetes.io/projected/858b4699-5ef5-46ca-bc2f-b80f700350e7-kube-api-access-r7qth\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.755018 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.754967    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.755075 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.755020    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.755182 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.755160    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.756995 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.756964    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.757240 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.757223    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858b4699-5ef5-46ca-bc2f-b80f700350e7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.762422 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.762396    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7qth\" (UniqueName: \"kubernetes.io/projected/858b4699-5ef5-46ca-bc2f-b80f700350e7-kube-api-access-r7qth\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:38.884837 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:38.884804    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:39.160548 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:39.160511    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"]
Sep 16 16:33:39.162463 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:33:39.162427    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858b4699_5ef5_46ca_bc2f_b80f700350e7.slice/crio-0352e03cc9d1db18cd165c399a3ab52b53e82e125a531682ba4a492b940e3f50 WatchSource:0}: Error finding container 0352e03cc9d1db18cd165c399a3ab52b53e82e125a531682ba4a492b940e3f50: Status 404 returned error can't find the container with id 0352e03cc9d1db18cd165c399a3ab52b53e82e125a531682ba4a492b940e3f50
Sep 16 16:33:39.715213 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:39.715179    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp" event={"ID":"858b4699-5ef5-46ca-bc2f-b80f700350e7","Type":"ContainerStarted","Data":"7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2"}
Sep 16 16:33:39.715629 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:39.715234    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp" event={"ID":"858b4699-5ef5-46ca-bc2f-b80f700350e7","Type":"ContainerStarted","Data":"0352e03cc9d1db18cd165c399a3ab52b53e82e125a531682ba4a492b940e3f50"}
Sep 16 16:33:39.732601 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:39.732551    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp" podStartSLOduration=1.732533908 podStartE2EDuration="1.732533908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:33:39.730738533 +0000 UTC m=+2616.250361311" watchObservedRunningTime="2026-09-16 16:33:39.732533908 +0000 UTC m=+2616.252156719"
Sep 16 16:33:39.885839 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:39.885803    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:48.885246 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:48.885205    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:48.897225 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:48.897193    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:33:49.776168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:33:49.776130    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:34:18.479252 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.479219    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"]
Sep 16 16:34:18.488598 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:34:18.488553    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:34:18.488598 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:34:18.488598 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:34:18.488598 ip-10-0-143-143 kubenswrapper[2574]:  > containerID="7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2" cmd=["/bin/sleep","15"]
Sep 16 16:34:18.488818 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.488617    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp" message="" err=<
Sep 16 16:34:18.488818 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:34:18.488818 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:34:18.488818 ip-10-0-143-143 kubenswrapper[2574]:  >
Sep 16 16:34:18.488818 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:34:18.488634    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:34:18.488818 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:34:18.488818 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:34:18.488818 ip-10-0-143-143 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp" podUID="858b4699-5ef5-46ca-bc2f-b80f700350e7" containerName="main" containerID="cri-o://7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2"
Sep 16 16:34:18.488818 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.488686    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp" podUID="858b4699-5ef5-46ca-bc2f-b80f700350e7" containerName="main" containerID="cri-o://7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2" gracePeriod=60
Sep 16 16:34:18.837022 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.836998    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:34:18.911298 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.911265    2574 generic.go:356] "Generic (PLEG): container finished" podID="858b4699-5ef5-46ca-bc2f-b80f700350e7" containerID="7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2" exitCode=0
Sep 16 16:34:18.911497 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.911322    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"
Sep 16 16:34:18.911497 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.911344    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp" event={"ID":"858b4699-5ef5-46ca-bc2f-b80f700350e7","Type":"ContainerDied","Data":"7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2"}
Sep 16 16:34:18.911497 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.911409    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp" event={"ID":"858b4699-5ef5-46ca-bc2f-b80f700350e7","Type":"ContainerDied","Data":"0352e03cc9d1db18cd165c399a3ab52b53e82e125a531682ba4a492b940e3f50"}
Sep 16 16:34:18.911497 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.911438    2574 scope.go:117] "RemoveContainer" containerID="7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2"
Sep 16 16:34:18.920234 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.920212    2574 scope.go:117] "RemoveContainer" containerID="7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2"
Sep 16 16:34:18.920502 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:34:18.920482    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2\": container with ID starting with 7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2 not found: ID does not exist" containerID="7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2"
Sep 16 16:34:18.920590 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.920528    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2"} err="failed to get container status \"7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2\": rpc error: code = NotFound desc = could not find container \"7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2\": container with ID starting with 7a85ca42f1ed9fbfe87291a22469d0a665f7bcab03e0680aaaf7e0b4ddddc9a2 not found: ID does not exist"
Sep 16 16:34:18.930860 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.930835    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7qth\" (UniqueName: \"kubernetes.io/projected/858b4699-5ef5-46ca-bc2f-b80f700350e7-kube-api-access-r7qth\") pod \"858b4699-5ef5-46ca-bc2f-b80f700350e7\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") "
Sep 16 16:34:18.930956 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.930893    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-home\") pod \"858b4699-5ef5-46ca-bc2f-b80f700350e7\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") "
Sep 16 16:34:18.930956 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.930927    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858b4699-5ef5-46ca-bc2f-b80f700350e7-tls-certs\") pod \"858b4699-5ef5-46ca-bc2f-b80f700350e7\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") "
Sep 16 16:34:18.931077 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.930985    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-tmp-dir\") pod \"858b4699-5ef5-46ca-bc2f-b80f700350e7\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") "
Sep 16 16:34:18.931077 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.931032    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-dshm\") pod \"858b4699-5ef5-46ca-bc2f-b80f700350e7\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") "
Sep 16 16:34:18.931180 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.931093    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-model-cache\") pod \"858b4699-5ef5-46ca-bc2f-b80f700350e7\" (UID: \"858b4699-5ef5-46ca-bc2f-b80f700350e7\") "
Sep 16 16:34:18.931253 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.931233    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-home" (OuterVolumeSpecName: "home") pod "858b4699-5ef5-46ca-bc2f-b80f700350e7" (UID: "858b4699-5ef5-46ca-bc2f-b80f700350e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:18.931311 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.931273    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "858b4699-5ef5-46ca-bc2f-b80f700350e7" (UID: "858b4699-5ef5-46ca-bc2f-b80f700350e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:18.931412 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.931394    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:18.931470 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.931417    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "858b4699-5ef5-46ca-bc2f-b80f700350e7" (UID: "858b4699-5ef5-46ca-bc2f-b80f700350e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:18.931470 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.931437    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:18.933159 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.933139    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/858b4699-5ef5-46ca-bc2f-b80f700350e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "858b4699-5ef5-46ca-bc2f-b80f700350e7" (UID: "858b4699-5ef5-46ca-bc2f-b80f700350e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:34:18.933262 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.933155    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-dshm" (OuterVolumeSpecName: "dshm") pod "858b4699-5ef5-46ca-bc2f-b80f700350e7" (UID: "858b4699-5ef5-46ca-bc2f-b80f700350e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:18.933262 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:18.933187    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858b4699-5ef5-46ca-bc2f-b80f700350e7-kube-api-access-r7qth" (OuterVolumeSpecName: "kube-api-access-r7qth") pod "858b4699-5ef5-46ca-bc2f-b80f700350e7" (UID: "858b4699-5ef5-46ca-bc2f-b80f700350e7"). InnerVolumeSpecName "kube-api-access-r7qth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:34:19.032055 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:19.031959    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858b4699-5ef5-46ca-bc2f-b80f700350e7-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:19.032055 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:19.031989    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:19.032055 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:19.031998    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858b4699-5ef5-46ca-bc2f-b80f700350e7-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:19.032055 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:19.032006    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7qth\" (UniqueName: \"kubernetes.io/projected/858b4699-5ef5-46ca-bc2f-b80f700350e7-kube-api-access-r7qth\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:19.232156 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:19.232123    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"]
Sep 16 16:34:19.238387 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:19.238361    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp"]
Sep 16 16:34:19.766857 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:19.766811    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hjtmp" podUID="858b4699-5ef5-46ca-bc2f-b80f700350e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 16:34:19.995061 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:19.995030    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="858b4699-5ef5-46ca-bc2f-b80f700350e7" path="/var/lib/kubelet/pods/858b4699-5ef5-46ca-bc2f-b80f700350e7/volumes"
Sep 16 16:34:34.749895 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:34.749851    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"]
Sep 16 16:34:35.449613 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:35.449574    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"]
Sep 16 16:34:36.153915 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:36.153887    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"]
Sep 16 16:34:49.771557 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:49.771489    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8" podUID="ca92abe9-6d0c-43f5-8061-4d0fd16255e5" containerName="main" containerID="cri-o://bdc908ae7c4d8b02987ef5532bd39228a27cefe1028c0b74988484903246da16" gracePeriod=45
Sep 16 16:34:50.015876 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.015851    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:34:50.076995 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.076901    2574 generic.go:356] "Generic (PLEG): container finished" podID="ca92abe9-6d0c-43f5-8061-4d0fd16255e5" containerID="bdc908ae7c4d8b02987ef5532bd39228a27cefe1028c0b74988484903246da16" exitCode=0
Sep 16 16:34:50.076995 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.076953    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"
Sep 16 16:34:50.077211 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.076991    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8" event={"ID":"ca92abe9-6d0c-43f5-8061-4d0fd16255e5","Type":"ContainerDied","Data":"bdc908ae7c4d8b02987ef5532bd39228a27cefe1028c0b74988484903246da16"}
Sep 16 16:34:50.077211 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.077048    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8" event={"ID":"ca92abe9-6d0c-43f5-8061-4d0fd16255e5","Type":"ContainerDied","Data":"3549093882cb9c31cacb4c851ddb38b49d739d85ec19e0314336faed5a80c4b4"}
Sep 16 16:34:50.077211 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.077075    2574 scope.go:117] "RemoveContainer" containerID="bdc908ae7c4d8b02987ef5532bd39228a27cefe1028c0b74988484903246da16"
Sep 16 16:34:50.085290 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.085273    2574 scope.go:117] "RemoveContainer" containerID="bdc908ae7c4d8b02987ef5532bd39228a27cefe1028c0b74988484903246da16"
Sep 16 16:34:50.085592 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:34:50.085570    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc908ae7c4d8b02987ef5532bd39228a27cefe1028c0b74988484903246da16\": container with ID starting with bdc908ae7c4d8b02987ef5532bd39228a27cefe1028c0b74988484903246da16 not found: ID does not exist" containerID="bdc908ae7c4d8b02987ef5532bd39228a27cefe1028c0b74988484903246da16"
Sep 16 16:34:50.085689 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.085598    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc908ae7c4d8b02987ef5532bd39228a27cefe1028c0b74988484903246da16"} err="failed to get container status \"bdc908ae7c4d8b02987ef5532bd39228a27cefe1028c0b74988484903246da16\": rpc error: code = NotFound desc = could not find container \"bdc908ae7c4d8b02987ef5532bd39228a27cefe1028c0b74988484903246da16\": container with ID starting with bdc908ae7c4d8b02987ef5532bd39228a27cefe1028c0b74988484903246da16 not found: ID does not exist"
Sep 16 16:34:50.110194 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.110159    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk5gk\" (UniqueName: \"kubernetes.io/projected/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-kube-api-access-pk5gk\") pod \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") "
Sep 16 16:34:50.110316 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.110229    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-tls-certs\") pod \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") "
Sep 16 16:34:50.110316 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.110254    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-model-cache\") pod \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") "
Sep 16 16:34:50.110408 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.110347    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-tmp-dir\") pod \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") "
Sep 16 16:34:50.110458 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.110408    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-dshm\") pod \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") "
Sep 16 16:34:50.110513 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.110459    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-home\") pod \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\" (UID: \"ca92abe9-6d0c-43f5-8061-4d0fd16255e5\") "
Sep 16 16:34:50.110613 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.110588    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca92abe9-6d0c-43f5-8061-4d0fd16255e5" (UID: "ca92abe9-6d0c-43f5-8061-4d0fd16255e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:50.110701 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.110680    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca92abe9-6d0c-43f5-8061-4d0fd16255e5" (UID: "ca92abe9-6d0c-43f5-8061-4d0fd16255e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:50.110841 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.110812    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-home" (OuterVolumeSpecName: "home") pod "ca92abe9-6d0c-43f5-8061-4d0fd16255e5" (UID: "ca92abe9-6d0c-43f5-8061-4d0fd16255e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:50.112609 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.111853    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:50.112609 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.111880    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:50.112609 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.111895    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:50.114558 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.113160    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca92abe9-6d0c-43f5-8061-4d0fd16255e5" (UID: "ca92abe9-6d0c-43f5-8061-4d0fd16255e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:34:50.117275 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.117238    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-dshm" (OuterVolumeSpecName: "dshm") pod "ca92abe9-6d0c-43f5-8061-4d0fd16255e5" (UID: "ca92abe9-6d0c-43f5-8061-4d0fd16255e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:50.120667 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.120429    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-kube-api-access-pk5gk" (OuterVolumeSpecName: "kube-api-access-pk5gk") pod "ca92abe9-6d0c-43f5-8061-4d0fd16255e5" (UID: "ca92abe9-6d0c-43f5-8061-4d0fd16255e5"). InnerVolumeSpecName "kube-api-access-pk5gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:34:50.212423 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.212395    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:50.212423 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.212421    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk5gk\" (UniqueName: \"kubernetes.io/projected/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-kube-api-access-pk5gk\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:50.212619 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.212431    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca92abe9-6d0c-43f5-8061-4d0fd16255e5-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:50.398632 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.398601    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"]
Sep 16 16:34:50.400359 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.400339    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-rdzz8"]
Sep 16 16:34:50.468315 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.468262    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69" podUID="4463d7cf-de55-4c56-9fce-f255a51d718c" containerName="main" containerID="cri-o://999cd451332fac1ddac5858247ba2c5d2f04436a1d05b93fa31e5a3ce8df06f7" gracePeriod=45
Sep 16 16:34:50.709446 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.709411    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:34:50.818782 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.818745    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4463d7cf-de55-4c56-9fce-f255a51d718c-tls-certs\") pod \"4463d7cf-de55-4c56-9fce-f255a51d718c\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") "
Sep 16 16:34:50.819181 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.818788    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-model-cache\") pod \"4463d7cf-de55-4c56-9fce-f255a51d718c\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") "
Sep 16 16:34:50.819181 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.818818    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6sxj\" (UniqueName: \"kubernetes.io/projected/4463d7cf-de55-4c56-9fce-f255a51d718c-kube-api-access-k6sxj\") pod \"4463d7cf-de55-4c56-9fce-f255a51d718c\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") "
Sep 16 16:34:50.819181 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.818964    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-dshm\") pod \"4463d7cf-de55-4c56-9fce-f255a51d718c\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") "
Sep 16 16:34:50.819181 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.819010    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-home\") pod \"4463d7cf-de55-4c56-9fce-f255a51d718c\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") "
Sep 16 16:34:50.819181 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.819083    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4463d7cf-de55-4c56-9fce-f255a51d718c" (UID: "4463d7cf-de55-4c56-9fce-f255a51d718c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:50.819181 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.819093    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-tmp-dir\") pod \"4463d7cf-de55-4c56-9fce-f255a51d718c\" (UID: \"4463d7cf-de55-4c56-9fce-f255a51d718c\") "
Sep 16 16:34:50.819453 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.819280    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-home" (OuterVolumeSpecName: "home") pod "4463d7cf-de55-4c56-9fce-f255a51d718c" (UID: "4463d7cf-de55-4c56-9fce-f255a51d718c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:50.819453 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.819289    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4463d7cf-de55-4c56-9fce-f255a51d718c" (UID: "4463d7cf-de55-4c56-9fce-f255a51d718c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:50.819557 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.819479    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:50.819557 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.819493    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:50.819557 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.819503    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:50.821001 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.820981    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4463d7cf-de55-4c56-9fce-f255a51d718c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4463d7cf-de55-4c56-9fce-f255a51d718c" (UID: "4463d7cf-de55-4c56-9fce-f255a51d718c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:34:50.821387 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.821365    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-dshm" (OuterVolumeSpecName: "dshm") pod "4463d7cf-de55-4c56-9fce-f255a51d718c" (UID: "4463d7cf-de55-4c56-9fce-f255a51d718c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:50.821440 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.821402    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4463d7cf-de55-4c56-9fce-f255a51d718c-kube-api-access-k6sxj" (OuterVolumeSpecName: "kube-api-access-k6sxj") pod "4463d7cf-de55-4c56-9fce-f255a51d718c" (UID: "4463d7cf-de55-4c56-9fce-f255a51d718c"). InnerVolumeSpecName "kube-api-access-k6sxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:34:50.920405 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.920366    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4463d7cf-de55-4c56-9fce-f255a51d718c-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:50.920405 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.920397    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6sxj\" (UniqueName: \"kubernetes.io/projected/4463d7cf-de55-4c56-9fce-f255a51d718c-kube-api-access-k6sxj\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:50.920405 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:50.920407    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4463d7cf-de55-4c56-9fce-f255a51d718c-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:51.084514 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.084422    2574 generic.go:356] "Generic (PLEG): container finished" podID="4463d7cf-de55-4c56-9fce-f255a51d718c" containerID="999cd451332fac1ddac5858247ba2c5d2f04436a1d05b93fa31e5a3ce8df06f7" exitCode=0
Sep 16 16:34:51.084514 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.084451    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69" event={"ID":"4463d7cf-de55-4c56-9fce-f255a51d718c","Type":"ContainerDied","Data":"999cd451332fac1ddac5858247ba2c5d2f04436a1d05b93fa31e5a3ce8df06f7"}
Sep 16 16:34:51.084514 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.084473    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"
Sep 16 16:34:51.084514 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.084514    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69" event={"ID":"4463d7cf-de55-4c56-9fce-f255a51d718c","Type":"ContainerDied","Data":"153122c9bbf5835e11f01621691566cad4a1c2a095c4307e2a93829de28d559c"}
Sep 16 16:34:51.084852 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.084551    2574 scope.go:117] "RemoveContainer" containerID="999cd451332fac1ddac5858247ba2c5d2f04436a1d05b93fa31e5a3ce8df06f7"
Sep 16 16:34:51.094242 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.094225    2574 scope.go:117] "RemoveContainer" containerID="999cd451332fac1ddac5858247ba2c5d2f04436a1d05b93fa31e5a3ce8df06f7"
Sep 16 16:34:51.094506 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:34:51.094488    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999cd451332fac1ddac5858247ba2c5d2f04436a1d05b93fa31e5a3ce8df06f7\": container with ID starting with 999cd451332fac1ddac5858247ba2c5d2f04436a1d05b93fa31e5a3ce8df06f7 not found: ID does not exist" containerID="999cd451332fac1ddac5858247ba2c5d2f04436a1d05b93fa31e5a3ce8df06f7"
Sep 16 16:34:51.094602 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.094538    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999cd451332fac1ddac5858247ba2c5d2f04436a1d05b93fa31e5a3ce8df06f7"} err="failed to get container status \"999cd451332fac1ddac5858247ba2c5d2f04436a1d05b93fa31e5a3ce8df06f7\": rpc error: code = NotFound desc = could not find container \"999cd451332fac1ddac5858247ba2c5d2f04436a1d05b93fa31e5a3ce8df06f7\": container with ID starting with 999cd451332fac1ddac5858247ba2c5d2f04436a1d05b93fa31e5a3ce8df06f7 not found: ID does not exist"
Sep 16 16:34:51.105039 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.105016    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"]
Sep 16 16:34:51.110588 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.110565    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-krq69"]
Sep 16 16:34:51.173403 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.173356    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s" podUID="4349f77a-20eb-4db3-abdd-512b5c77135c" containerName="main" containerID="cri-o://89a1ac7d12f047473238c0855e1d9c439475fcdf75962b1510778f4901f653e5" gracePeriod=45
Sep 16 16:34:51.421919 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.421894    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:34:51.526070 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.526035    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-home\") pod \"4349f77a-20eb-4db3-abdd-512b5c77135c\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") "
Sep 16 16:34:51.526280 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.526081    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-dshm\") pod \"4349f77a-20eb-4db3-abdd-512b5c77135c\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") "
Sep 16 16:34:51.526280 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.526130    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k7p7\" (UniqueName: \"kubernetes.io/projected/4349f77a-20eb-4db3-abdd-512b5c77135c-kube-api-access-8k7p7\") pod \"4349f77a-20eb-4db3-abdd-512b5c77135c\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") "
Sep 16 16:34:51.526280 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.526184    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-model-cache\") pod \"4349f77a-20eb-4db3-abdd-512b5c77135c\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") "
Sep 16 16:34:51.526280 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.526230    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-tmp-dir\") pod \"4349f77a-20eb-4db3-abdd-512b5c77135c\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") "
Sep 16 16:34:51.526492 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.526328    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4349f77a-20eb-4db3-abdd-512b5c77135c-tls-certs\") pod \"4349f77a-20eb-4db3-abdd-512b5c77135c\" (UID: \"4349f77a-20eb-4db3-abdd-512b5c77135c\") "
Sep 16 16:34:51.526492 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.526343    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-home" (OuterVolumeSpecName: "home") pod "4349f77a-20eb-4db3-abdd-512b5c77135c" (UID: "4349f77a-20eb-4db3-abdd-512b5c77135c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:51.526639 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.526509    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4349f77a-20eb-4db3-abdd-512b5c77135c" (UID: "4349f77a-20eb-4db3-abdd-512b5c77135c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:51.526639 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.526563    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4349f77a-20eb-4db3-abdd-512b5c77135c" (UID: "4349f77a-20eb-4db3-abdd-512b5c77135c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:51.526747 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.526663    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:51.526747 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.526674    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:51.526747 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.526685    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:51.528225 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.528201    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-dshm" (OuterVolumeSpecName: "dshm") pod "4349f77a-20eb-4db3-abdd-512b5c77135c" (UID: "4349f77a-20eb-4db3-abdd-512b5c77135c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:34:51.528329 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.528310    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4349f77a-20eb-4db3-abdd-512b5c77135c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4349f77a-20eb-4db3-abdd-512b5c77135c" (UID: "4349f77a-20eb-4db3-abdd-512b5c77135c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:34:51.528383 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.528366    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4349f77a-20eb-4db3-abdd-512b5c77135c-kube-api-access-8k7p7" (OuterVolumeSpecName: "kube-api-access-8k7p7") pod "4349f77a-20eb-4db3-abdd-512b5c77135c" (UID: "4349f77a-20eb-4db3-abdd-512b5c77135c"). InnerVolumeSpecName "kube-api-access-8k7p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:34:51.628182 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.628150    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4349f77a-20eb-4db3-abdd-512b5c77135c-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:51.628182 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.628179    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4349f77a-20eb-4db3-abdd-512b5c77135c-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:51.628382 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.628189    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k7p7\" (UniqueName: \"kubernetes.io/projected/4349f77a-20eb-4db3-abdd-512b5c77135c-kube-api-access-8k7p7\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:34:51.994227 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.994145    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4463d7cf-de55-4c56-9fce-f255a51d718c" path="/var/lib/kubelet/pods/4463d7cf-de55-4c56-9fce-f255a51d718c/volumes"
Sep 16 16:34:51.994582 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:51.994569    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca92abe9-6d0c-43f5-8061-4d0fd16255e5" path="/var/lib/kubelet/pods/ca92abe9-6d0c-43f5-8061-4d0fd16255e5/volumes"
Sep 16 16:34:52.021635 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:34:52.021595    2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4349f77a_20eb_4db3_abdd_512b5c77135c.slice/crio-2569bbf687bca9a12d29f34b4db68c426962f79ad238251d0b5e647f8958a0e1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4349f77a_20eb_4db3_abdd_512b5c77135c.slice\": RecentStats: unable to find data in memory cache]"
Sep 16 16:34:52.091040 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:52.091008    2574 generic.go:356] "Generic (PLEG): container finished" podID="4349f77a-20eb-4db3-abdd-512b5c77135c" containerID="89a1ac7d12f047473238c0855e1d9c439475fcdf75962b1510778f4901f653e5" exitCode=0
Sep 16 16:34:52.091198 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:52.091054    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"
Sep 16 16:34:52.091198 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:52.091061    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s" event={"ID":"4349f77a-20eb-4db3-abdd-512b5c77135c","Type":"ContainerDied","Data":"89a1ac7d12f047473238c0855e1d9c439475fcdf75962b1510778f4901f653e5"}
Sep 16 16:34:52.091198 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:52.091102    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s" event={"ID":"4349f77a-20eb-4db3-abdd-512b5c77135c","Type":"ContainerDied","Data":"2569bbf687bca9a12d29f34b4db68c426962f79ad238251d0b5e647f8958a0e1"}
Sep 16 16:34:52.091198 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:52.091124    2574 scope.go:117] "RemoveContainer" containerID="89a1ac7d12f047473238c0855e1d9c439475fcdf75962b1510778f4901f653e5"
Sep 16 16:34:52.099504 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:52.099486    2574 scope.go:117] "RemoveContainer" containerID="89a1ac7d12f047473238c0855e1d9c439475fcdf75962b1510778f4901f653e5"
Sep 16 16:34:52.099774 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:34:52.099758    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a1ac7d12f047473238c0855e1d9c439475fcdf75962b1510778f4901f653e5\": container with ID starting with 89a1ac7d12f047473238c0855e1d9c439475fcdf75962b1510778f4901f653e5 not found: ID does not exist" containerID="89a1ac7d12f047473238c0855e1d9c439475fcdf75962b1510778f4901f653e5"
Sep 16 16:34:52.099831 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:52.099785    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a1ac7d12f047473238c0855e1d9c439475fcdf75962b1510778f4901f653e5"} err="failed to get container status \"89a1ac7d12f047473238c0855e1d9c439475fcdf75962b1510778f4901f653e5\": rpc error: code = NotFound desc = could not find container \"89a1ac7d12f047473238c0855e1d9c439475fcdf75962b1510778f4901f653e5\": container with ID starting with 89a1ac7d12f047473238c0855e1d9c439475fcdf75962b1510778f4901f653e5 not found: ID does not exist"
Sep 16 16:34:52.107013 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:52.106990    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"]
Sep 16 16:34:52.113471 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:52.113450    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-rk26s"]
Sep 16 16:34:53.994727 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:34:53.994699    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4349f77a-20eb-4db3-abdd-512b5c77135c" path="/var/lib/kubelet/pods/4349f77a-20eb-4db3-abdd-512b5c77135c/volumes"
Sep 16 16:35:03.898638 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.898602    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"]
Sep 16 16:35:03.899038 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.898985    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca92abe9-6d0c-43f5-8061-4d0fd16255e5" containerName="main"
Sep 16 16:35:03.899038 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.898998    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca92abe9-6d0c-43f5-8061-4d0fd16255e5" containerName="main"
Sep 16 16:35:03.899038 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.899015    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="858b4699-5ef5-46ca-bc2f-b80f700350e7" containerName="main"
Sep 16 16:35:03.899038 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.899021    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="858b4699-5ef5-46ca-bc2f-b80f700350e7" containerName="main"
Sep 16 16:35:03.899038 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.899032    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4349f77a-20eb-4db3-abdd-512b5c77135c" containerName="main"
Sep 16 16:35:03.899038 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.899037    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4349f77a-20eb-4db3-abdd-512b5c77135c" containerName="main"
Sep 16 16:35:03.899254 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.899055    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4463d7cf-de55-4c56-9fce-f255a51d718c" containerName="main"
Sep 16 16:35:03.899254 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.899061    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4463d7cf-de55-4c56-9fce-f255a51d718c" containerName="main"
Sep 16 16:35:03.899254 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.899112    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4463d7cf-de55-4c56-9fce-f255a51d718c" containerName="main"
Sep 16 16:35:03.899254 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.899120    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="858b4699-5ef5-46ca-bc2f-b80f700350e7" containerName="main"
Sep 16 16:35:03.899254 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.899126    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca92abe9-6d0c-43f5-8061-4d0fd16255e5" containerName="main"
Sep 16 16:35:03.899254 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.899133    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4349f77a-20eb-4db3-abdd-512b5c77135c" containerName="main"
Sep 16 16:35:03.903725 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.903700    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:03.906039 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.906016    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 16 16:35:03.906811 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.906791    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 16 16:35:03.907018 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.907003    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-kwcnd\""
Sep 16 16:35:03.907119 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.907098    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 16 16:35:03.913643 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.913619    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"]
Sep 16 16:35:03.937185 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.937158    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-model-cache\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:03.937327 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.937194    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:03.937327 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.937262    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6nfk\" (UniqueName: \"kubernetes.io/projected/48f984c4-0bf3-43e0-837c-e8641ba89201-kube-api-access-b6nfk\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:03.937327 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.937288    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-dshm\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:03.937427 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.937332    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-home\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:03.937427 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:03.937362    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f984c4-0bf3-43e0-837c-e8641ba89201-tls-certs\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:04.037990 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.037954    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-model-cache\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:04.037990 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.037995    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:04.038198 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.038030    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6nfk\" (UniqueName: \"kubernetes.io/projected/48f984c4-0bf3-43e0-837c-e8641ba89201-kube-api-access-b6nfk\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:04.038198 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.038056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-dshm\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:04.038198 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.038072    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-home\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:04.038198 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.038086    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f984c4-0bf3-43e0-837c-e8641ba89201-tls-certs\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:04.038428 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.038398    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-model-cache\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:04.038481 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.038429    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:04.038481 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.038468    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-home\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:04.040340 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.040320    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-dshm\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:04.040607 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.040590    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f984c4-0bf3-43e0-837c-e8641ba89201-tls-certs\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:04.045186 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.045160    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6nfk\" (UniqueName: \"kubernetes.io/projected/48f984c4-0bf3-43e0-837c-e8641ba89201-kube-api-access-b6nfk\") pod \"late-v1-kserve-56cbc8456c-zv6gn\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:04.216856 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.216761    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:04.341786 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.341756    2574 scope.go:117] "RemoveContainer" containerID="b299fe2b177ef8f71d10c4730db6da13c1aefb1cb1f96c7621bebca10cb01a5b"
Sep 16 16:35:04.350190 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.350172    2574 scope.go:117] "RemoveContainer" containerID="bb67fea401e7694cfd06a632f5e634606f7263a6d37cdc71969570d531ffffaa"
Sep 16 16:35:04.490866 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:04.490839    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"]
Sep 16 16:35:04.493683 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:35:04.493649    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f984c4_0bf3_43e0_837c_e8641ba89201.slice/crio-afdee3cc23b64415eca029fa10c816931c500526205d83be750a28c104625e1a WatchSource:0}: Error finding container afdee3cc23b64415eca029fa10c816931c500526205d83be750a28c104625e1a: Status 404 returned error can't find the container with id afdee3cc23b64415eca029fa10c816931c500526205d83be750a28c104625e1a
Sep 16 16:35:05.157055 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:05.157013    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn" event={"ID":"48f984c4-0bf3-43e0-837c-e8641ba89201","Type":"ContainerStarted","Data":"16cb43eaaf089fb268b19f52023991e656149c6d289d30a0859e774ebe1bb968"}
Sep 16 16:35:05.157578 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:05.157073    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn" event={"ID":"48f984c4-0bf3-43e0-837c-e8641ba89201","Type":"ContainerStarted","Data":"afdee3cc23b64415eca029fa10c816931c500526205d83be750a28c104625e1a"}
Sep 16 16:35:05.174007 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:05.173958    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn" podStartSLOduration=2.173941371 podStartE2EDuration="2.173941371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:35:05.172651463 +0000 UTC m=+2701.692274275" watchObservedRunningTime="2026-09-16 16:35:05.173941371 +0000 UTC m=+2701.693564145"
Sep 16 16:35:05.217903 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:05.217868    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:14.217821 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:14.217773    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:14.229809 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:14.229778    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:15.214395 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:15.214361    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:35:30.567175 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.567137    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"]
Sep 16 16:35:30.570766 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.570747    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.573772 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.573749    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 16 16:35:30.581797 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.581764    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"]
Sep 16 16:35:30.692911 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.692873    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-tmp-dir\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.693109 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.692949    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-dshm\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.693109 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.692978    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-model-cache\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.693109 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.693011    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d460765-ffe5-4410-b923-999a1f870fa9-tls-certs\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.693109 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.693086    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chlf6\" (UniqueName: \"kubernetes.io/projected/0d460765-ffe5-4410-b923-999a1f870fa9-kube-api-access-chlf6\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.693272 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.693117    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-home\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.794541 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.794489    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-dshm\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.794739 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.794554    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-model-cache\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.794739 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.794586    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d460765-ffe5-4410-b923-999a1f870fa9-tls-certs\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.794739 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.794612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chlf6\" (UniqueName: \"kubernetes.io/projected/0d460765-ffe5-4410-b923-999a1f870fa9-kube-api-access-chlf6\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.794739 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.794627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-home\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.794739 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.794651    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-tmp-dir\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.795050 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.795018    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-tmp-dir\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.795155 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.795054    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-model-cache\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.795155 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.795069    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-home\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.796800 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.796780    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-dshm\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.797117 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.797101    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d460765-ffe5-4410-b923-999a1f870fa9-tls-certs\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.801652 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.801629    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chlf6\" (UniqueName: \"kubernetes.io/projected/0d460765-ffe5-4410-b923-999a1f870fa9-kube-api-access-chlf6\") pod \"late-v2-kserve-86766546f-hhg7t\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:30.882966 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:30.882943    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:31.152975 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:31.152951    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"]
Sep 16 16:35:31.155441 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:35:31.155402    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d460765_ffe5_4410_b923_999a1f870fa9.slice/crio-25a16e31a76b57f5580e5acd9966660228f53aeb4e584b2621df65b214a63d86 WatchSource:0}: Error finding container 25a16e31a76b57f5580e5acd9966660228f53aeb4e584b2621df65b214a63d86: Status 404 returned error can't find the container with id 25a16e31a76b57f5580e5acd9966660228f53aeb4e584b2621df65b214a63d86
Sep 16 16:35:31.280335 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:31.280306    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t" event={"ID":"0d460765-ffe5-4410-b923-999a1f870fa9","Type":"ContainerStarted","Data":"49649274007b5028fa12a3d2c192ce9b3dd2c88434a2e666a7bcc15e4739428b"}
Sep 16 16:35:31.280456 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:31.280361    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t" event={"ID":"0d460765-ffe5-4410-b923-999a1f870fa9","Type":"ContainerStarted","Data":"25a16e31a76b57f5580e5acd9966660228f53aeb4e584b2621df65b214a63d86"}
Sep 16 16:35:31.298927 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:31.298879    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t" podStartSLOduration=1.298865434 podStartE2EDuration="1.298865434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:35:31.297455432 +0000 UTC m=+2727.817078208" watchObservedRunningTime="2026-09-16 16:35:31.298865434 +0000 UTC m=+2727.818488209"
Sep 16 16:35:31.883969 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:31.883933    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:40.883194 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:40.883150    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:40.895783 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:40.895757    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:35:41.339934 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:35:41.339854    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:36:13.630679 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:13.630641    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"]
Sep 16 16:36:13.635484 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:13.635459    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"]
Sep 16 16:36:28.649714 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.649629    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn" podUID="48f984c4-0bf3-43e0-837c-e8641ba89201" containerName="main" containerID="cri-o://16cb43eaaf089fb268b19f52023991e656149c6d289d30a0859e774ebe1bb968" gracePeriod=45
Sep 16 16:36:28.653350 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.653320    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t" podUID="0d460765-ffe5-4410-b923-999a1f870fa9" containerName="main" containerID="cri-o://49649274007b5028fa12a3d2c192ce9b3dd2c88434a2e666a7bcc15e4739428b" gracePeriod=45
Sep 16 16:36:28.903104 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.903048    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:36:28.924862 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.924835    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:36:28.934071 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.934046    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f984c4-0bf3-43e0-837c-e8641ba89201-tls-certs\") pod \"48f984c4-0bf3-43e0-837c-e8641ba89201\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") "
Sep 16 16:36:28.934178 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.934097    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-home\") pod \"48f984c4-0bf3-43e0-837c-e8641ba89201\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") "
Sep 16 16:36:28.934178 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.934130    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6nfk\" (UniqueName: \"kubernetes.io/projected/48f984c4-0bf3-43e0-837c-e8641ba89201-kube-api-access-b6nfk\") pod \"48f984c4-0bf3-43e0-837c-e8641ba89201\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") "
Sep 16 16:36:28.934178 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.934175    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-model-cache\") pod \"48f984c4-0bf3-43e0-837c-e8641ba89201\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") "
Sep 16 16:36:28.934344 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.934208    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-dshm\") pod \"48f984c4-0bf3-43e0-837c-e8641ba89201\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") "
Sep 16 16:36:28.934344 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.934266    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-tmp-dir\") pod \"48f984c4-0bf3-43e0-837c-e8641ba89201\" (UID: \"48f984c4-0bf3-43e0-837c-e8641ba89201\") "
Sep 16 16:36:28.934586 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.934495    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-model-cache" (OuterVolumeSpecName: "model-cache") pod "48f984c4-0bf3-43e0-837c-e8641ba89201" (UID: "48f984c4-0bf3-43e0-837c-e8641ba89201"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:36:28.934702 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.934589    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48f984c4-0bf3-43e0-837c-e8641ba89201" (UID: "48f984c4-0bf3-43e0-837c-e8641ba89201"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:36:28.934765 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.934735    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-home" (OuterVolumeSpecName: "home") pod "48f984c4-0bf3-43e0-837c-e8641ba89201" (UID: "48f984c4-0bf3-43e0-837c-e8641ba89201"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:36:28.936389 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.936361    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f984c4-0bf3-43e0-837c-e8641ba89201-kube-api-access-b6nfk" (OuterVolumeSpecName: "kube-api-access-b6nfk") pod "48f984c4-0bf3-43e0-837c-e8641ba89201" (UID: "48f984c4-0bf3-43e0-837c-e8641ba89201"). InnerVolumeSpecName "kube-api-access-b6nfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:36:28.936570 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.936554    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48f984c4-0bf3-43e0-837c-e8641ba89201-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48f984c4-0bf3-43e0-837c-e8641ba89201" (UID: "48f984c4-0bf3-43e0-837c-e8641ba89201"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:36:28.936691 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:28.936666    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-dshm" (OuterVolumeSpecName: "dshm") pod "48f984c4-0bf3-43e0-837c-e8641ba89201" (UID: "48f984c4-0bf3-43e0-837c-e8641ba89201"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:36:29.035551 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.035509    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chlf6\" (UniqueName: \"kubernetes.io/projected/0d460765-ffe5-4410-b923-999a1f870fa9-kube-api-access-chlf6\") pod \"0d460765-ffe5-4410-b923-999a1f870fa9\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") "
Sep 16 16:36:29.035717 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.035577    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-home\") pod \"0d460765-ffe5-4410-b923-999a1f870fa9\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") "
Sep 16 16:36:29.035717 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.035608    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-dshm\") pod \"0d460765-ffe5-4410-b923-999a1f870fa9\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") "
Sep 16 16:36:29.035717 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.035623    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-model-cache\") pod \"0d460765-ffe5-4410-b923-999a1f870fa9\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") "
Sep 16 16:36:29.035717 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.035639    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-tmp-dir\") pod \"0d460765-ffe5-4410-b923-999a1f870fa9\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") "
Sep 16 16:36:29.035717 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.035662    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d460765-ffe5-4410-b923-999a1f870fa9-tls-certs\") pod \"0d460765-ffe5-4410-b923-999a1f870fa9\" (UID: \"0d460765-ffe5-4410-b923-999a1f870fa9\") "
Sep 16 16:36:29.035982 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.035819    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-home" (OuterVolumeSpecName: "home") pod "0d460765-ffe5-4410-b923-999a1f870fa9" (UID: "0d460765-ffe5-4410-b923-999a1f870fa9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:36:29.035982 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.035907    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d460765-ffe5-4410-b923-999a1f870fa9" (UID: "0d460765-ffe5-4410-b923-999a1f870fa9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:36:29.035982 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.035925    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d460765-ffe5-4410-b923-999a1f870fa9" (UID: "0d460765-ffe5-4410-b923-999a1f870fa9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:36:29.036140 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.036036    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:36:29.036140 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.036054    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:36:29.036140 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.036068    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:36:29.036140 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.036080    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:36:29.036140 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.036091    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f984c4-0bf3-43e0-837c-e8641ba89201-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:36:29.036140 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.036104    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:36:29.036140 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.036118    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f984c4-0bf3-43e0-837c-e8641ba89201-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:36:29.036140 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.036131    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:36:29.036399 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.036143    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6nfk\" (UniqueName: \"kubernetes.io/projected/48f984c4-0bf3-43e0-837c-e8641ba89201-kube-api-access-b6nfk\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:36:29.037662 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.037638    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-dshm" (OuterVolumeSpecName: "dshm") pod "0d460765-ffe5-4410-b923-999a1f870fa9" (UID: "0d460765-ffe5-4410-b923-999a1f870fa9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:36:29.037771 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.037680    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d460765-ffe5-4410-b923-999a1f870fa9-kube-api-access-chlf6" (OuterVolumeSpecName: "kube-api-access-chlf6") pod "0d460765-ffe5-4410-b923-999a1f870fa9" (UID: "0d460765-ffe5-4410-b923-999a1f870fa9"). InnerVolumeSpecName "kube-api-access-chlf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:36:29.037771 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.037711    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d460765-ffe5-4410-b923-999a1f870fa9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d460765-ffe5-4410-b923-999a1f870fa9" (UID: "0d460765-ffe5-4410-b923-999a1f870fa9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:36:29.137267 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.137242    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chlf6\" (UniqueName: \"kubernetes.io/projected/0d460765-ffe5-4410-b923-999a1f870fa9-kube-api-access-chlf6\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:36:29.137267 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.137265    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d460765-ffe5-4410-b923-999a1f870fa9-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:36:29.137419 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.137273    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d460765-ffe5-4410-b923-999a1f870fa9-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:36:29.555378 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.555344    2574 generic.go:356] "Generic (PLEG): container finished" podID="0d460765-ffe5-4410-b923-999a1f870fa9" containerID="49649274007b5028fa12a3d2c192ce9b3dd2c88434a2e666a7bcc15e4739428b" exitCode=0
Sep 16 16:36:29.555573 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.555391    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"
Sep 16 16:36:29.555573 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.555428    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t" event={"ID":"0d460765-ffe5-4410-b923-999a1f870fa9","Type":"ContainerDied","Data":"49649274007b5028fa12a3d2c192ce9b3dd2c88434a2e666a7bcc15e4739428b"}
Sep 16 16:36:29.555573 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.555486    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t" event={"ID":"0d460765-ffe5-4410-b923-999a1f870fa9","Type":"ContainerDied","Data":"25a16e31a76b57f5580e5acd9966660228f53aeb4e584b2621df65b214a63d86"}
Sep 16 16:36:29.555573 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.555528    2574 scope.go:117] "RemoveContainer" containerID="49649274007b5028fa12a3d2c192ce9b3dd2c88434a2e666a7bcc15e4739428b"
Sep 16 16:36:29.556867 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.556837    2574 generic.go:356] "Generic (PLEG): container finished" podID="48f984c4-0bf3-43e0-837c-e8641ba89201" containerID="16cb43eaaf089fb268b19f52023991e656149c6d289d30a0859e774ebe1bb968" exitCode=0
Sep 16 16:36:29.556970 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.556880    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"
Sep 16 16:36:29.556970 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.556928    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn" event={"ID":"48f984c4-0bf3-43e0-837c-e8641ba89201","Type":"ContainerDied","Data":"16cb43eaaf089fb268b19f52023991e656149c6d289d30a0859e774ebe1bb968"}
Sep 16 16:36:29.557058 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.556983    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn" event={"ID":"48f984c4-0bf3-43e0-837c-e8641ba89201","Type":"ContainerDied","Data":"afdee3cc23b64415eca029fa10c816931c500526205d83be750a28c104625e1a"}
Sep 16 16:36:29.564447 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.564318    2574 scope.go:117] "RemoveContainer" containerID="49649274007b5028fa12a3d2c192ce9b3dd2c88434a2e666a7bcc15e4739428b"
Sep 16 16:36:29.564699 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:36:29.564671    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49649274007b5028fa12a3d2c192ce9b3dd2c88434a2e666a7bcc15e4739428b\": container with ID starting with 49649274007b5028fa12a3d2c192ce9b3dd2c88434a2e666a7bcc15e4739428b not found: ID does not exist" containerID="49649274007b5028fa12a3d2c192ce9b3dd2c88434a2e666a7bcc15e4739428b"
Sep 16 16:36:29.564768 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.564705    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49649274007b5028fa12a3d2c192ce9b3dd2c88434a2e666a7bcc15e4739428b"} err="failed to get container status \"49649274007b5028fa12a3d2c192ce9b3dd2c88434a2e666a7bcc15e4739428b\": rpc error: code = NotFound desc = could not find container \"49649274007b5028fa12a3d2c192ce9b3dd2c88434a2e666a7bcc15e4739428b\": container with ID starting with 49649274007b5028fa12a3d2c192ce9b3dd2c88434a2e666a7bcc15e4739428b not found: ID does not exist"
Sep 16 16:36:29.564768 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.564724    2574 scope.go:117] "RemoveContainer" containerID="16cb43eaaf089fb268b19f52023991e656149c6d289d30a0859e774ebe1bb968"
Sep 16 16:36:29.572182 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.572167    2574 scope.go:117] "RemoveContainer" containerID="16cb43eaaf089fb268b19f52023991e656149c6d289d30a0859e774ebe1bb968"
Sep 16 16:36:29.572432 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:36:29.572413    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16cb43eaaf089fb268b19f52023991e656149c6d289d30a0859e774ebe1bb968\": container with ID starting with 16cb43eaaf089fb268b19f52023991e656149c6d289d30a0859e774ebe1bb968 not found: ID does not exist" containerID="16cb43eaaf089fb268b19f52023991e656149c6d289d30a0859e774ebe1bb968"
Sep 16 16:36:29.572490 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.572437    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16cb43eaaf089fb268b19f52023991e656149c6d289d30a0859e774ebe1bb968"} err="failed to get container status \"16cb43eaaf089fb268b19f52023991e656149c6d289d30a0859e774ebe1bb968\": rpc error: code = NotFound desc = could not find container \"16cb43eaaf089fb268b19f52023991e656149c6d289d30a0859e774ebe1bb968\": container with ID starting with 16cb43eaaf089fb268b19f52023991e656149c6d289d30a0859e774ebe1bb968 not found: ID does not exist"
Sep 16 16:36:29.578432 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.578410    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"]
Sep 16 16:36:29.581987 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.581968    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-hhg7t"]
Sep 16 16:36:29.592760 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.592732    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"]
Sep 16 16:36:29.596911 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.596892    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zv6gn"]
Sep 16 16:36:29.994912 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.994875    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d460765-ffe5-4410-b923-999a1f870fa9" path="/var/lib/kubelet/pods/0d460765-ffe5-4410-b923-999a1f870fa9/volumes"
Sep 16 16:36:29.995438 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:29.995417    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48f984c4-0bf3-43e0-837c-e8641ba89201" path="/var/lib/kubelet/pods/48f984c4-0bf3-43e0-837c-e8641ba89201/volumes"
Sep 16 16:36:37.070278 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.070247    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"]
Sep 16 16:36:37.070657 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.070641    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f984c4-0bf3-43e0-837c-e8641ba89201" containerName="main"
Sep 16 16:36:37.070657 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.070653    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f984c4-0bf3-43e0-837c-e8641ba89201" containerName="main"
Sep 16 16:36:37.070730 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.070678    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d460765-ffe5-4410-b923-999a1f870fa9" containerName="main"
Sep 16 16:36:37.070730 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.070684    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d460765-ffe5-4410-b923-999a1f870fa9" containerName="main"
Sep 16 16:36:37.070791 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.070735    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d460765-ffe5-4410-b923-999a1f870fa9" containerName="main"
Sep 16 16:36:37.070791 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.070763    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="48f984c4-0bf3-43e0-837c-e8641ba89201" containerName="main"
Sep 16 16:36:37.075025 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.075000    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.077277 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.077255    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-5cf5j\""
Sep 16 16:36:37.077409 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.077322    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 16 16:36:37.077907 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.077888    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 16 16:36:37.077982 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.077910    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 16 16:36:37.082324 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.082303    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"]
Sep 16 16:36:37.209598 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.209560    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.209598 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.209608    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-dshm\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.209820 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.209684    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-home\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.209820 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.209724    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzqlx\" (UniqueName: \"kubernetes.io/projected/21364716-c6a3-4b40-9238-199c321c7d93-kube-api-access-lzqlx\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.209820 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.209775    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21364716-c6a3-4b40-9238-199c321c7d93-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.209936 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.209826    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.310509 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.310458    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21364716-c6a3-4b40-9238-199c321c7d93-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.310509 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.310512    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.310776 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.310573    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.310776 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.310596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-dshm\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.310776 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.310649    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-home\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.310776 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.310689    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzqlx\" (UniqueName: \"kubernetes.io/projected/21364716-c6a3-4b40-9238-199c321c7d93-kube-api-access-lzqlx\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.311022 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.310997    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.311100 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.311041    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.311100 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.311072    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-home\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.312942 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.312920    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-dshm\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.313369 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.313346    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21364716-c6a3-4b40-9238-199c321c7d93-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.317811 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.317791    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzqlx\" (UniqueName: \"kubernetes.io/projected/21364716-c6a3-4b40-9238-199c321c7d93-kube-api-access-lzqlx\") pod \"delw-v1-kserve-6cf8cbb847-6lcgt\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.386657 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.386629    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:37.662891 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:37.662810    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"]
Sep 16 16:36:37.666274 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:36:37.666246    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21364716_c6a3_4b40_9238_199c321c7d93.slice/crio-4fde90fd9cf24810cec98077e71bd7ff8d3980cc2dd9ff6428cf3b2453318025 WatchSource:0}: Error finding container 4fde90fd9cf24810cec98077e71bd7ff8d3980cc2dd9ff6428cf3b2453318025: Status 404 returned error can't find the container with id 4fde90fd9cf24810cec98077e71bd7ff8d3980cc2dd9ff6428cf3b2453318025
Sep 16 16:36:38.606695 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:38.606662    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt" event={"ID":"21364716-c6a3-4b40-9238-199c321c7d93","Type":"ContainerStarted","Data":"5967299b9710980dedbf2b5239949d64f00803570c69a0cbe6bf85f2793b445f"}
Sep 16 16:36:38.607083 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:38.606713    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt" event={"ID":"21364716-c6a3-4b40-9238-199c321c7d93","Type":"ContainerStarted","Data":"4fde90fd9cf24810cec98077e71bd7ff8d3980cc2dd9ff6428cf3b2453318025"}
Sep 16 16:36:38.625785 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:38.625736    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt" podStartSLOduration=1.62572011 podStartE2EDuration="1.62572011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:36:38.624209906 +0000 UTC m=+2795.143832681" watchObservedRunningTime="2026-09-16 16:36:38.62572011 +0000 UTC m=+2795.145342885"
Sep 16 16:36:39.387041 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:39.386992    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:43.065870 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.065837    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"]
Sep 16 16:36:43.069293 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.069275    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.071411 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.071391    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 16 16:36:43.077556 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.077535    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"]
Sep 16 16:36:43.169425 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.169390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.169425 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.169426    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc4a512-a967-4efe-82f3-351d37a2649e-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.169650 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.169476    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-749nn\" (UniqueName: \"kubernetes.io/projected/8fc4a512-a967-4efe-82f3-351d37a2649e-kube-api-access-749nn\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.169650 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.169503    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-home\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.169650 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.169592    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.169766 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.169675    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.270170 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.270138    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.270336 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.270175    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc4a512-a967-4efe-82f3-351d37a2649e-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.270336 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.270213    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-749nn\" (UniqueName: \"kubernetes.io/projected/8fc4a512-a967-4efe-82f3-351d37a2649e-kube-api-access-749nn\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.270336 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.270232    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-home\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.270336 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.270251    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.270336 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.270301    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.270620 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.270592    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.270660 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.270627    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-home\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.270702 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.270686    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.272480 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.272454    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.272872 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.272852    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc4a512-a967-4efe-82f3-351d37a2649e-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.277320 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.277298    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-749nn\" (UniqueName: \"kubernetes.io/projected/8fc4a512-a967-4efe-82f3-351d37a2649e-kube-api-access-749nn\") pod \"delw-v2-kserve-5b84cc7ddf-68ngx\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.380648 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.380607    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:43.650316 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:43.650290    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"]
Sep 16 16:36:43.652533 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:36:43.652492    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc4a512_a967_4efe_82f3_351d37a2649e.slice/crio-3c60f48d38a8513ac7535c2347be081b4be35437f4092310ee9271ac39d435ac WatchSource:0}: Error finding container 3c60f48d38a8513ac7535c2347be081b4be35437f4092310ee9271ac39d435ac: Status 404 returned error can't find the container with id 3c60f48d38a8513ac7535c2347be081b4be35437f4092310ee9271ac39d435ac
Sep 16 16:36:44.642069 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:44.642027    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx" event={"ID":"8fc4a512-a967-4efe-82f3-351d37a2649e","Type":"ContainerStarted","Data":"bbf436a8d5af89522df7291eb3d7cb7d522755f7c3c0a77a09a2acc87025fd9b"}
Sep 16 16:36:44.642509 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:44.642114    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx" event={"ID":"8fc4a512-a967-4efe-82f3-351d37a2649e","Type":"ContainerStarted","Data":"3c60f48d38a8513ac7535c2347be081b4be35437f4092310ee9271ac39d435ac"}
Sep 16 16:36:44.659377 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:44.659331    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx" podStartSLOduration=1.659317241 podStartE2EDuration="1.659317241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:36:44.657495481 +0000 UTC m=+2801.177118257" watchObservedRunningTime="2026-09-16 16:36:44.659317241 +0000 UTC m=+2801.178940017"
Sep 16 16:36:45.381037 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:45.380999    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:47.387111 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:47.387064    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:47.399581 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:47.399555    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:47.667483 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:47.667399    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:36:53.381584 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:53.381548    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:53.393300 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:53.393274    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:36:53.697611 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:36:53.697535    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:37:26.451640 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.451606    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"]
Sep 16 16:37:26.455180 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.455159    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.458155 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.458127    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 16:37:26.458252 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.458162    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 16:37:26.458252 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.458194    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-27sbj\""
Sep 16 16:37:26.458252 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.458166    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 16:37:26.466024 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.466003    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"]
Sep 16 16:37:26.557038 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.557008    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.557038 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.557041    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.557276 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.557083    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49wp9\" (UniqueName: \"kubernetes.io/projected/3787a054-fb6e-4786-b8b4-d041864a111a-kube-api-access-49wp9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.557276 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.557139    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.557276 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.557213    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3787a054-fb6e-4786-b8b4-d041864a111a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.557276 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.557238    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.657933 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.657897    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.658123 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.657971    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3787a054-fb6e-4786-b8b4-d041864a111a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.658123 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.657992    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.658123 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.658025    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.658123 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.658043    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.658123 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.658083    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49wp9\" (UniqueName: \"kubernetes.io/projected/3787a054-fb6e-4786-b8b4-d041864a111a-kube-api-access-49wp9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.658375 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.658282    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.658375 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.658330    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.658621 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.658589    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.660486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.660457    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.660589 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.660539    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3787a054-fb6e-4786-b8b4-d041864a111a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.665408 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.665387    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49wp9\" (UniqueName: \"kubernetes.io/projected/3787a054-fb6e-4786-b8b4-d041864a111a-kube-api-access-49wp9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:26.766860 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:26.766766    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:27.039349 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:27.039323    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"]
Sep 16 16:37:27.041315 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:37:27.041280    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3787a054_fb6e_4786_b8b4_d041864a111a.slice/crio-89e43371d016c1e380c680b26b414bcd26bd43716fde245f96968dd38abb8c92 WatchSource:0}: Error finding container 89e43371d016c1e380c680b26b414bcd26bd43716fde245f96968dd38abb8c92: Status 404 returned error can't find the container with id 89e43371d016c1e380c680b26b414bcd26bd43716fde245f96968dd38abb8c92
Sep 16 16:37:27.850280 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:27.850243    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp" event={"ID":"3787a054-fb6e-4786-b8b4-d041864a111a","Type":"ContainerStarted","Data":"19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec"}
Sep 16 16:37:27.850793 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:27.850300    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp" event={"ID":"3787a054-fb6e-4786-b8b4-d041864a111a","Type":"ContainerStarted","Data":"89e43371d016c1e380c680b26b414bcd26bd43716fde245f96968dd38abb8c92"}
Sep 16 16:37:27.868421 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:27.868373    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp" podStartSLOduration=1.868358952 podStartE2EDuration="1.868358952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:37:27.866862139 +0000 UTC m=+2844.386484919" watchObservedRunningTime="2026-09-16 16:37:27.868358952 +0000 UTC m=+2844.387981728"
Sep 16 16:37:28.767008 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:28.766977    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:31.748738 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:31.748701    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"]
Sep 16 16:37:36.767504 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:36.767463    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:36.779299 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:36.779272    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:36.904030 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:36.904003    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:37:46.767705 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:46.767646    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx" podUID="8fc4a512-a967-4efe-82f3-351d37a2649e" containerName="main" containerID="cri-o://bbf436a8d5af89522df7291eb3d7cb7d522755f7c3c0a77a09a2acc87025fd9b" gracePeriod=45
Sep 16 16:37:46.949716 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:46.949682    2574 generic.go:356] "Generic (PLEG): container finished" podID="8fc4a512-a967-4efe-82f3-351d37a2649e" containerID="bbf436a8d5af89522df7291eb3d7cb7d522755f7c3c0a77a09a2acc87025fd9b" exitCode=0
Sep 16 16:37:46.949885 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:46.949740    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx" event={"ID":"8fc4a512-a967-4efe-82f3-351d37a2649e","Type":"ContainerDied","Data":"bbf436a8d5af89522df7291eb3d7cb7d522755f7c3c0a77a09a2acc87025fd9b"}
Sep 16 16:37:47.008894 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.008868    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:37:47.151008 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.150976    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-tmp-dir\") pod \"8fc4a512-a967-4efe-82f3-351d37a2649e\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") "
Sep 16 16:37:47.151205 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.151025    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-model-cache\") pod \"8fc4a512-a967-4efe-82f3-351d37a2649e\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") "
Sep 16 16:37:47.151205 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.151068    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-749nn\" (UniqueName: \"kubernetes.io/projected/8fc4a512-a967-4efe-82f3-351d37a2649e-kube-api-access-749nn\") pod \"8fc4a512-a967-4efe-82f3-351d37a2649e\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") "
Sep 16 16:37:47.151205 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.151085    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-dshm\") pod \"8fc4a512-a967-4efe-82f3-351d37a2649e\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") "
Sep 16 16:37:47.151205 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.151113    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc4a512-a967-4efe-82f3-351d37a2649e-tls-certs\") pod \"8fc4a512-a967-4efe-82f3-351d37a2649e\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") "
Sep 16 16:37:47.151205 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.151176    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-home\") pod \"8fc4a512-a967-4efe-82f3-351d37a2649e\" (UID: \"8fc4a512-a967-4efe-82f3-351d37a2649e\") "
Sep 16 16:37:47.151473 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.151283    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fc4a512-a967-4efe-82f3-351d37a2649e" (UID: "8fc4a512-a967-4efe-82f3-351d37a2649e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:37:47.151473 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.151390    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fc4a512-a967-4efe-82f3-351d37a2649e" (UID: "8fc4a512-a967-4efe-82f3-351d37a2649e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:37:47.151473 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.151410    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-home" (OuterVolumeSpecName: "home") pod "8fc4a512-a967-4efe-82f3-351d37a2649e" (UID: "8fc4a512-a967-4efe-82f3-351d37a2649e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:37:47.151640 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.151499    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:37:47.151640 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.151546    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:37:47.151640 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.151565    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:37:47.153402 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.153381    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc4a512-a967-4efe-82f3-351d37a2649e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fc4a512-a967-4efe-82f3-351d37a2649e" (UID: "8fc4a512-a967-4efe-82f3-351d37a2649e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:37:47.153467 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.153402    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-dshm" (OuterVolumeSpecName: "dshm") pod "8fc4a512-a967-4efe-82f3-351d37a2649e" (UID: "8fc4a512-a967-4efe-82f3-351d37a2649e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:37:47.153510 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.153473    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc4a512-a967-4efe-82f3-351d37a2649e-kube-api-access-749nn" (OuterVolumeSpecName: "kube-api-access-749nn") pod "8fc4a512-a967-4efe-82f3-351d37a2649e" (UID: "8fc4a512-a967-4efe-82f3-351d37a2649e"). InnerVolumeSpecName "kube-api-access-749nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:37:47.252301 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.252266    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-749nn\" (UniqueName: \"kubernetes.io/projected/8fc4a512-a967-4efe-82f3-351d37a2649e-kube-api-access-749nn\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:37:47.252301 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.252295    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc4a512-a967-4efe-82f3-351d37a2649e-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:37:47.252301 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.252306    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc4a512-a967-4efe-82f3-351d37a2649e-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:37:47.956618 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.956589    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"
Sep 16 16:37:47.956618 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.956610    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx" event={"ID":"8fc4a512-a967-4efe-82f3-351d37a2649e","Type":"ContainerDied","Data":"3c60f48d38a8513ac7535c2347be081b4be35437f4092310ee9271ac39d435ac"}
Sep 16 16:37:47.957110 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.956675    2574 scope.go:117] "RemoveContainer" containerID="bbf436a8d5af89522df7291eb3d7cb7d522755f7c3c0a77a09a2acc87025fd9b"
Sep 16 16:37:47.979326 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.979296    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"]
Sep 16 16:37:47.988158 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.988133    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-68ngx"]
Sep 16 16:37:47.995539 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:47.995493    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fc4a512-a967-4efe-82f3-351d37a2649e" path="/var/lib/kubelet/pods/8fc4a512-a967-4efe-82f3-351d37a2649e/volumes"
Sep 16 16:37:52.473763 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:37:52.473729    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"]
Sep 16 16:38:07.492422 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.492360    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt" podUID="21364716-c6a3-4b40-9238-199c321c7d93" containerName="main" containerID="cri-o://5967299b9710980dedbf2b5239949d64f00803570c69a0cbe6bf85f2793b445f" gracePeriod=45
Sep 16 16:38:07.657688 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.657646    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt" podUID="21364716-c6a3-4b40-9238-199c321c7d93" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 16:38:07.737622 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.737594    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:38:07.822314 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.822232    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzqlx\" (UniqueName: \"kubernetes.io/projected/21364716-c6a3-4b40-9238-199c321c7d93-kube-api-access-lzqlx\") pod \"21364716-c6a3-4b40-9238-199c321c7d93\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") "
Sep 16 16:38:07.822314 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.822298    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21364716-c6a3-4b40-9238-199c321c7d93-tls-certs\") pod \"21364716-c6a3-4b40-9238-199c321c7d93\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") "
Sep 16 16:38:07.822551 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.822352    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-dshm\") pod \"21364716-c6a3-4b40-9238-199c321c7d93\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") "
Sep 16 16:38:07.822551 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.822403    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-model-cache\") pod \"21364716-c6a3-4b40-9238-199c321c7d93\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") "
Sep 16 16:38:07.822551 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.822432    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-home\") pod \"21364716-c6a3-4b40-9238-199c321c7d93\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") "
Sep 16 16:38:07.822551 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.822452    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-tmp-dir\") pod \"21364716-c6a3-4b40-9238-199c321c7d93\" (UID: \"21364716-c6a3-4b40-9238-199c321c7d93\") "
Sep 16 16:38:07.822766 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.822701    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-model-cache" (OuterVolumeSpecName: "model-cache") pod "21364716-c6a3-4b40-9238-199c321c7d93" (UID: "21364716-c6a3-4b40-9238-199c321c7d93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:38:07.822871 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.822854    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:38:07.822928 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.822870    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-home" (OuterVolumeSpecName: "home") pod "21364716-c6a3-4b40-9238-199c321c7d93" (UID: "21364716-c6a3-4b40-9238-199c321c7d93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:38:07.822972 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.822913    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21364716-c6a3-4b40-9238-199c321c7d93" (UID: "21364716-c6a3-4b40-9238-199c321c7d93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:38:07.824443 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.824421    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-dshm" (OuterVolumeSpecName: "dshm") pod "21364716-c6a3-4b40-9238-199c321c7d93" (UID: "21364716-c6a3-4b40-9238-199c321c7d93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:38:07.824540 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.824453    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21364716-c6a3-4b40-9238-199c321c7d93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21364716-c6a3-4b40-9238-199c321c7d93" (UID: "21364716-c6a3-4b40-9238-199c321c7d93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:38:07.824594 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.824577    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21364716-c6a3-4b40-9238-199c321c7d93-kube-api-access-lzqlx" (OuterVolumeSpecName: "kube-api-access-lzqlx") pod "21364716-c6a3-4b40-9238-199c321c7d93" (UID: "21364716-c6a3-4b40-9238-199c321c7d93"). InnerVolumeSpecName "kube-api-access-lzqlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:38:07.923957 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.923933    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:38:07.923957 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.923955    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:38:07.924080 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.923966    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzqlx\" (UniqueName: \"kubernetes.io/projected/21364716-c6a3-4b40-9238-199c321c7d93-kube-api-access-lzqlx\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:38:07.924080 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.923975    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21364716-c6a3-4b40-9238-199c321c7d93-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:38:07.924080 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:07.923983    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21364716-c6a3-4b40-9238-199c321c7d93-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:38:08.054727 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:08.054696    2574 generic.go:356] "Generic (PLEG): container finished" podID="21364716-c6a3-4b40-9238-199c321c7d93" containerID="5967299b9710980dedbf2b5239949d64f00803570c69a0cbe6bf85f2793b445f" exitCode=0
Sep 16 16:38:08.054886 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:08.054739    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"
Sep 16 16:38:08.054886 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:08.054786    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt" event={"ID":"21364716-c6a3-4b40-9238-199c321c7d93","Type":"ContainerDied","Data":"5967299b9710980dedbf2b5239949d64f00803570c69a0cbe6bf85f2793b445f"}
Sep 16 16:38:08.054886 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:08.054843    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt" event={"ID":"21364716-c6a3-4b40-9238-199c321c7d93","Type":"ContainerDied","Data":"4fde90fd9cf24810cec98077e71bd7ff8d3980cc2dd9ff6428cf3b2453318025"}
Sep 16 16:38:08.054886 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:08.054871    2574 scope.go:117] "RemoveContainer" containerID="5967299b9710980dedbf2b5239949d64f00803570c69a0cbe6bf85f2793b445f"
Sep 16 16:38:08.063899 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:08.063880    2574 scope.go:117] "RemoveContainer" containerID="5967299b9710980dedbf2b5239949d64f00803570c69a0cbe6bf85f2793b445f"
Sep 16 16:38:08.064181 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:38:08.064158    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5967299b9710980dedbf2b5239949d64f00803570c69a0cbe6bf85f2793b445f\": container with ID starting with 5967299b9710980dedbf2b5239949d64f00803570c69a0cbe6bf85f2793b445f not found: ID does not exist" containerID="5967299b9710980dedbf2b5239949d64f00803570c69a0cbe6bf85f2793b445f"
Sep 16 16:38:08.064314 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:08.064187    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5967299b9710980dedbf2b5239949d64f00803570c69a0cbe6bf85f2793b445f"} err="failed to get container status \"5967299b9710980dedbf2b5239949d64f00803570c69a0cbe6bf85f2793b445f\": rpc error: code = NotFound desc = could not find container \"5967299b9710980dedbf2b5239949d64f00803570c69a0cbe6bf85f2793b445f\": container with ID starting with 5967299b9710980dedbf2b5239949d64f00803570c69a0cbe6bf85f2793b445f not found: ID does not exist"
Sep 16 16:38:08.071237 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:08.071212    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"]
Sep 16 16:38:08.076620 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:08.076569    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-6lcgt"]
Sep 16 16:38:09.564747 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.564718    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"]
Sep 16 16:38:09.565153 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.565135    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21364716-c6a3-4b40-9238-199c321c7d93" containerName="main"
Sep 16 16:38:09.565201 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.565157    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="21364716-c6a3-4b40-9238-199c321c7d93" containerName="main"
Sep 16 16:38:09.565201 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.565171    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc4a512-a967-4efe-82f3-351d37a2649e" containerName="main"
Sep 16 16:38:09.565201 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.565177    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc4a512-a967-4efe-82f3-351d37a2649e" containerName="main"
Sep 16 16:38:09.565303 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.565251    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc4a512-a967-4efe-82f3-351d37a2649e" containerName="main"
Sep 16 16:38:09.565303 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.565260    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="21364716-c6a3-4b40-9238-199c321c7d93" containerName="main"
Sep 16 16:38:09.569739 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.569719    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.572709 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.572676    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 16 16:38:09.572709 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.572696    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Sep 16 16:38:09.572709 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.572710    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 16 16:38:09.572955 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.572684    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-pnh7g\""
Sep 16 16:38:09.577543 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.577489    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"]
Sep 16 16:38:09.639501 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.639475    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.639647 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.639512    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.639647 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.639617    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-dshm\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.639720 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.639673    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8279891d-d0c2-4570-a8a9-28940539bbb0-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.639720 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.639702    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-home\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.639786 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.639722    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2f52\" (UniqueName: \"kubernetes.io/projected/8279891d-d0c2-4570-a8a9-28940539bbb0-kube-api-access-k2f52\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.740973 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.740939    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2f52\" (UniqueName: \"kubernetes.io/projected/8279891d-d0c2-4570-a8a9-28940539bbb0-kube-api-access-k2f52\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.741121 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.741007    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.741121 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.741039    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.741121 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.741080    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-dshm\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.741265 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.741143    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8279891d-d0c2-4570-a8a9-28940539bbb0-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.741265 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.741172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-home\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.741477 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.741452    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.741580 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.741478    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.741580 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.741505    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-home\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.743391 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.743371    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-dshm\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.743706 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.743686    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8279891d-d0c2-4570-a8a9-28940539bbb0-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.748490 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.748472    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2f52\" (UniqueName: \"kubernetes.io/projected/8279891d-d0c2-4570-a8a9-28940539bbb0-kube-api-access-k2f52\") pod \"roll-v1-kserve-69cf8bcfc-rbcqd\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.880691 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.880655    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:09.994626 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:09.994590    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21364716-c6a3-4b40-9238-199c321c7d93" path="/var/lib/kubelet/pods/21364716-c6a3-4b40-9238-199c321c7d93/volumes"
Sep 16 16:38:10.151401 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:10.151374    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"]
Sep 16 16:38:10.153251 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:38:10.153228    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8279891d_d0c2_4570_a8a9_28940539bbb0.slice/crio-268e0b5903e57ac72991ac9f77643ced921ca94c31024a7099cb51c275f1c307 WatchSource:0}: Error finding container 268e0b5903e57ac72991ac9f77643ced921ca94c31024a7099cb51c275f1c307: Status 404 returned error can't find the container with id 268e0b5903e57ac72991ac9f77643ced921ca94c31024a7099cb51c275f1c307
Sep 16 16:38:11.070592 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:11.070557    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd" event={"ID":"8279891d-d0c2-4570-a8a9-28940539bbb0","Type":"ContainerStarted","Data":"fa6ac5bd9b3b00ec9c9dc61cc97f4f801bf94d5a69d919836fb29624c1d0e1ab"}
Sep 16 16:38:11.070946 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:11.070606    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd" event={"ID":"8279891d-d0c2-4570-a8a9-28940539bbb0","Type":"ContainerStarted","Data":"268e0b5903e57ac72991ac9f77643ced921ca94c31024a7099cb51c275f1c307"}
Sep 16 16:38:11.087341 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:11.087294    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd" podStartSLOduration=2.087278818 podStartE2EDuration="2.087278818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:38:11.08587813 +0000 UTC m=+2887.605500899" watchObservedRunningTime="2026-09-16 16:38:11.087278818 +0000 UTC m=+2887.606901594"
Sep 16 16:38:11.881270 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:11.881237    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:19.881677 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:19.881636    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:19.893225 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:19.893191    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:20.126923 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:20.126894    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:38:26.867836 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:26.867801    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"]
Sep 16 16:38:26.871745 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:26.871718    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:26.873708 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:26.873688    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 16 16:38:26.881814 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:26.881793    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"]
Sep 16 16:38:26.994106 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:26.994078    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:26.994106 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:26.994109    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj8xv\" (UniqueName: \"kubernetes.io/projected/38d9a68e-f401-47d6-acdb-f07936772cf3-kube-api-access-cj8xv\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:26.994330 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:26.994130    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:26.994373 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:26.994338    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-dshm\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:26.994428 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:26.994411    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d9a68e-f401-47d6-acdb-f07936772cf3-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:26.994490 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:26.994475    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-home\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:27.095342 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:27.095305    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:27.095342 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:27.095344    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj8xv\" (UniqueName: \"kubernetes.io/projected/38d9a68e-f401-47d6-acdb-f07936772cf3-kube-api-access-cj8xv\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:27.095626 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:27.095377    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:27.095626 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:27.095472    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-dshm\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:27.095626 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:27.095512    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d9a68e-f401-47d6-acdb-f07936772cf3-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:27.095626 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:27.095581    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-home\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:27.095815 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:27.095748    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:27.095815 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:27.095777    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:27.095943 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:27.095921    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-home\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:27.097732 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:27.097709    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-dshm\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:27.098050 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:27.098033    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d9a68e-f401-47d6-acdb-f07936772cf3-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:27.102823 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:27.102803    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj8xv\" (UniqueName: \"kubernetes.io/projected/38d9a68e-f401-47d6-acdb-f07936772cf3-kube-api-access-cj8xv\") pod \"roll-v2-kserve-d4dbf9d67-fshcd\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:27.183741 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:27.183670    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:27.453325 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:27.453298    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"]
Sep 16 16:38:27.455097 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:38:27.455069    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d9a68e_f401_47d6_acdb_f07936772cf3.slice/crio-cd4cd86d757d625b9122c68aa8e4902fa3b78004f8ece02c09a48a7f63ce174c WatchSource:0}: Error finding container cd4cd86d757d625b9122c68aa8e4902fa3b78004f8ece02c09a48a7f63ce174c: Status 404 returned error can't find the container with id cd4cd86d757d625b9122c68aa8e4902fa3b78004f8ece02c09a48a7f63ce174c
Sep 16 16:38:28.170149 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.170102    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd" event={"ID":"38d9a68e-f401-47d6-acdb-f07936772cf3","Type":"ContainerStarted","Data":"7abe6ff130ebd29438560084cced89dffb548107fc176a02c623098063f8495c"}
Sep 16 16:38:28.170644 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.170178    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd" event={"ID":"38d9a68e-f401-47d6-acdb-f07936772cf3","Type":"ContainerStarted","Data":"cd4cd86d757d625b9122c68aa8e4902fa3b78004f8ece02c09a48a7f63ce174c"}
Sep 16 16:38:28.184686 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.184654    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:28.201436 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.201389    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd" podStartSLOduration=2.201374855 podStartE2EDuration="2.201374855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:38:28.199347298 +0000 UTC m=+2904.718970075" watchObservedRunningTime="2026-09-16 16:38:28.201374855 +0000 UTC m=+2904.720997632"
Sep 16 16:38:28.221153 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.221123    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"]
Sep 16 16:38:28.232771 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:38:28.232725    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 16:38:28.232771 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:38:28.232771 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:38:28.232771 ip-10-0-143-143 kubenswrapper[2574]:  > containerID="19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec" cmd=["/bin/sleep","15"]
Sep 16 16:38:28.233069 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.232781    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp" message="" err=<
Sep 16 16:38:28.233069 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:38:28.233069 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:38:28.233069 ip-10-0-143-143 kubenswrapper[2574]:  >
Sep 16 16:38:28.233069 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:38:28.232793    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 16:38:28.233069 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 16:38:28.233069 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 16:38:28.233069 ip-10-0-143-143 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp" podUID="3787a054-fb6e-4786-b8b4-d041864a111a" containerName="main" containerID="cri-o://19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec"
Sep 16 16:38:28.233069 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.232831    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp" podUID="3787a054-fb6e-4786-b8b4-d041864a111a" containerName="main" containerID="cri-o://19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec" gracePeriod=60
Sep 16 16:38:28.580601 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.580574    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:38:28.717994 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.717925    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3787a054-fb6e-4786-b8b4-d041864a111a-tls-certs\") pod \"3787a054-fb6e-4786-b8b4-d041864a111a\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") "
Sep 16 16:38:28.717994 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.717983    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-tmp-dir\") pod \"3787a054-fb6e-4786-b8b4-d041864a111a\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") "
Sep 16 16:38:28.718192 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.718001    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-model-cache\") pod \"3787a054-fb6e-4786-b8b4-d041864a111a\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") "
Sep 16 16:38:28.718192 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.718019    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-dshm\") pod \"3787a054-fb6e-4786-b8b4-d041864a111a\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") "
Sep 16 16:38:28.718192 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.718038    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49wp9\" (UniqueName: \"kubernetes.io/projected/3787a054-fb6e-4786-b8b4-d041864a111a-kube-api-access-49wp9\") pod \"3787a054-fb6e-4786-b8b4-d041864a111a\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") "
Sep 16 16:38:28.718192 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.718077    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-home\") pod \"3787a054-fb6e-4786-b8b4-d041864a111a\" (UID: \"3787a054-fb6e-4786-b8b4-d041864a111a\") "
Sep 16 16:38:28.718391 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.718267    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-model-cache" (OuterVolumeSpecName: "model-cache") pod "3787a054-fb6e-4786-b8b4-d041864a111a" (UID: "3787a054-fb6e-4786-b8b4-d041864a111a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:38:28.718391 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.718324    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3787a054-fb6e-4786-b8b4-d041864a111a" (UID: "3787a054-fb6e-4786-b8b4-d041864a111a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:38:28.718494 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.718463    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-home" (OuterVolumeSpecName: "home") pod "3787a054-fb6e-4786-b8b4-d041864a111a" (UID: "3787a054-fb6e-4786-b8b4-d041864a111a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:38:28.720086 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.720058    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-dshm" (OuterVolumeSpecName: "dshm") pod "3787a054-fb6e-4786-b8b4-d041864a111a" (UID: "3787a054-fb6e-4786-b8b4-d041864a111a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:38:28.720181 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.720133    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3787a054-fb6e-4786-b8b4-d041864a111a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3787a054-fb6e-4786-b8b4-d041864a111a" (UID: "3787a054-fb6e-4786-b8b4-d041864a111a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:38:28.720181 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.720131    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3787a054-fb6e-4786-b8b4-d041864a111a-kube-api-access-49wp9" (OuterVolumeSpecName: "kube-api-access-49wp9") pod "3787a054-fb6e-4786-b8b4-d041864a111a" (UID: "3787a054-fb6e-4786-b8b4-d041864a111a"). InnerVolumeSpecName "kube-api-access-49wp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:38:28.819010 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.818980    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3787a054-fb6e-4786-b8b4-d041864a111a-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:38:28.819010 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.819005    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:38:28.819160 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.819019    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:38:28.819160 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.819030    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:38:28.819160 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.819042    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49wp9\" (UniqueName: \"kubernetes.io/projected/3787a054-fb6e-4786-b8b4-d041864a111a-kube-api-access-49wp9\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:38:28.819160 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:28.819055    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3787a054-fb6e-4786-b8b4-d041864a111a-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:38:29.176324 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:29.176296    2574 generic.go:356] "Generic (PLEG): container finished" podID="3787a054-fb6e-4786-b8b4-d041864a111a" containerID="19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec" exitCode=0
Sep 16 16:38:29.176704 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:29.176335    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"
Sep 16 16:38:29.176704 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:29.176378    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp" event={"ID":"3787a054-fb6e-4786-b8b4-d041864a111a","Type":"ContainerDied","Data":"19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec"}
Sep 16 16:38:29.176704 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:29.176432    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp" event={"ID":"3787a054-fb6e-4786-b8b4-d041864a111a","Type":"ContainerDied","Data":"89e43371d016c1e380c680b26b414bcd26bd43716fde245f96968dd38abb8c92"}
Sep 16 16:38:29.176704 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:29.176457    2574 scope.go:117] "RemoveContainer" containerID="19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec"
Sep 16 16:38:29.185805 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:29.185791    2574 scope.go:117] "RemoveContainer" containerID="19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec"
Sep 16 16:38:29.186047 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:38:29.186028    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec\": container with ID starting with 19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec not found: ID does not exist" containerID="19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec"
Sep 16 16:38:29.186111 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:29.186058    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec"} err="failed to get container status \"19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec\": rpc error: code = NotFound desc = could not find container \"19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec\": container with ID starting with 19596e2af935ded7db90dac6422b63ef1414c217080031d4d2ecd7e9136137ec not found: ID does not exist"
Sep 16 16:38:29.196960 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:29.196939    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"]
Sep 16 16:38:29.202837 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:29.202816    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xj9qp"]
Sep 16 16:38:29.994646 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:29.994615    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3787a054-fb6e-4786-b8b4-d041864a111a" path="/var/lib/kubelet/pods/3787a054-fb6e-4786-b8b4-d041864a111a/volumes"
Sep 16 16:38:37.184052 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:37.184019    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:37.195619 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:37.195589    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:38:37.228315 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:38:37.228286    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:40:06.620648 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:06.620612    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"]
Sep 16 16:40:07.064449 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:07.064370    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"]
Sep 16 16:40:21.639921 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.639858    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd" podUID="38d9a68e-f401-47d6-acdb-f07936772cf3" containerName="main" containerID="cri-o://7abe6ff130ebd29438560084cced89dffb548107fc176a02c623098063f8495c" gracePeriod=45
Sep 16 16:40:21.884150 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.884123    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:40:21.990033 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.989959    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d9a68e-f401-47d6-acdb-f07936772cf3-tls-certs\") pod \"38d9a68e-f401-47d6-acdb-f07936772cf3\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") "
Sep 16 16:40:21.990033 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.989997    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-model-cache\") pod \"38d9a68e-f401-47d6-acdb-f07936772cf3\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") "
Sep 16 16:40:21.990033 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.990016    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-home\") pod \"38d9a68e-f401-47d6-acdb-f07936772cf3\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") "
Sep 16 16:40:21.990324 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.990057    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-dshm\") pod \"38d9a68e-f401-47d6-acdb-f07936772cf3\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") "
Sep 16 16:40:21.990324 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.990127    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-tmp-dir\") pod \"38d9a68e-f401-47d6-acdb-f07936772cf3\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") "
Sep 16 16:40:21.990324 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.990169    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj8xv\" (UniqueName: \"kubernetes.io/projected/38d9a68e-f401-47d6-acdb-f07936772cf3-kube-api-access-cj8xv\") pod \"38d9a68e-f401-47d6-acdb-f07936772cf3\" (UID: \"38d9a68e-f401-47d6-acdb-f07936772cf3\") "
Sep 16 16:40:21.990324 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.990284    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-model-cache" (OuterVolumeSpecName: "model-cache") pod "38d9a68e-f401-47d6-acdb-f07936772cf3" (UID: "38d9a68e-f401-47d6-acdb-f07936772cf3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:40:21.990324 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.990299    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-home" (OuterVolumeSpecName: "home") pod "38d9a68e-f401-47d6-acdb-f07936772cf3" (UID: "38d9a68e-f401-47d6-acdb-f07936772cf3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:40:21.990613 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.990435    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38d9a68e-f401-47d6-acdb-f07936772cf3" (UID: "38d9a68e-f401-47d6-acdb-f07936772cf3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:40:21.990613 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.990574    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:40:21.990613 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.990598    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:40:21.990726 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.990616    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:40:21.992428 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.992402    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d9a68e-f401-47d6-acdb-f07936772cf3-kube-api-access-cj8xv" (OuterVolumeSpecName: "kube-api-access-cj8xv") pod "38d9a68e-f401-47d6-acdb-f07936772cf3" (UID: "38d9a68e-f401-47d6-acdb-f07936772cf3"). InnerVolumeSpecName "kube-api-access-cj8xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:40:21.992546 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.992427    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-dshm" (OuterVolumeSpecName: "dshm") pod "38d9a68e-f401-47d6-acdb-f07936772cf3" (UID: "38d9a68e-f401-47d6-acdb-f07936772cf3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:40:21.992546 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:21.992448    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38d9a68e-f401-47d6-acdb-f07936772cf3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38d9a68e-f401-47d6-acdb-f07936772cf3" (UID: "38d9a68e-f401-47d6-acdb-f07936772cf3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:40:22.082819 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.082762    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd" podUID="8279891d-d0c2-4570-a8a9-28940539bbb0" containerName="main" containerID="cri-o://fa6ac5bd9b3b00ec9c9dc61cc97f4f801bf94d5a69d919836fb29624c1d0e1ab" gracePeriod=45
Sep 16 16:40:22.091617 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.091582    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj8xv\" (UniqueName: \"kubernetes.io/projected/38d9a68e-f401-47d6-acdb-f07936772cf3-kube-api-access-cj8xv\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:40:22.091617 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.091614    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d9a68e-f401-47d6-acdb-f07936772cf3-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:40:22.091752 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.091623    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d9a68e-f401-47d6-acdb-f07936772cf3-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:40:22.116573 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.116545    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd" podUID="8279891d-d0c2-4570-a8a9-28940539bbb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 16 16:40:22.318688 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.318664    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:40:22.496205 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.496111    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-tmp-dir\") pod \"8279891d-d0c2-4570-a8a9-28940539bbb0\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") "
Sep 16 16:40:22.496205 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.496169    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-home\") pod \"8279891d-d0c2-4570-a8a9-28940539bbb0\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") "
Sep 16 16:40:22.496205 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.496194    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-model-cache\") pod \"8279891d-d0c2-4570-a8a9-28940539bbb0\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") "
Sep 16 16:40:22.496489 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.496246    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8279891d-d0c2-4570-a8a9-28940539bbb0-tls-certs\") pod \"8279891d-d0c2-4570-a8a9-28940539bbb0\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") "
Sep 16 16:40:22.496489 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.496266    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2f52\" (UniqueName: \"kubernetes.io/projected/8279891d-d0c2-4570-a8a9-28940539bbb0-kube-api-access-k2f52\") pod \"8279891d-d0c2-4570-a8a9-28940539bbb0\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") "
Sep 16 16:40:22.496489 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.496287    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-dshm\") pod \"8279891d-d0c2-4570-a8a9-28940539bbb0\" (UID: \"8279891d-d0c2-4570-a8a9-28940539bbb0\") "
Sep 16 16:40:22.496489 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.496427    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8279891d-d0c2-4570-a8a9-28940539bbb0" (UID: "8279891d-d0c2-4570-a8a9-28940539bbb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:40:22.496696 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.496495    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-home" (OuterVolumeSpecName: "home") pod "8279891d-d0c2-4570-a8a9-28940539bbb0" (UID: "8279891d-d0c2-4570-a8a9-28940539bbb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:40:22.496696 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.496542    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-model-cache" (OuterVolumeSpecName: "model-cache") pod "8279891d-d0c2-4570-a8a9-28940539bbb0" (UID: "8279891d-d0c2-4570-a8a9-28940539bbb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:40:22.496696 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.496608    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:40:22.496696 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.496627    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:40:22.498385 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.498361    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-dshm" (OuterVolumeSpecName: "dshm") pod "8279891d-d0c2-4570-a8a9-28940539bbb0" (UID: "8279891d-d0c2-4570-a8a9-28940539bbb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:40:22.498477 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.498396    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8279891d-d0c2-4570-a8a9-28940539bbb0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8279891d-d0c2-4570-a8a9-28940539bbb0" (UID: "8279891d-d0c2-4570-a8a9-28940539bbb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:40:22.498477 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.498442    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8279891d-d0c2-4570-a8a9-28940539bbb0-kube-api-access-k2f52" (OuterVolumeSpecName: "kube-api-access-k2f52") pod "8279891d-d0c2-4570-a8a9-28940539bbb0" (UID: "8279891d-d0c2-4570-a8a9-28940539bbb0"). InnerVolumeSpecName "kube-api-access-k2f52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:40:22.597208 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.597180    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8279891d-d0c2-4570-a8a9-28940539bbb0-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:40:22.597208 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.597205    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2f52\" (UniqueName: \"kubernetes.io/projected/8279891d-d0c2-4570-a8a9-28940539bbb0-kube-api-access-k2f52\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:40:22.597208 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.597214    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:40:22.597405 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.597224    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8279891d-d0c2-4570-a8a9-28940539bbb0-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:40:22.749153 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.749076    2574 generic.go:356] "Generic (PLEG): container finished" podID="8279891d-d0c2-4570-a8a9-28940539bbb0" containerID="fa6ac5bd9b3b00ec9c9dc61cc97f4f801bf94d5a69d919836fb29624c1d0e1ab" exitCode=0
Sep 16 16:40:22.749153 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.749117    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"
Sep 16 16:40:22.749636 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.749115    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd" event={"ID":"8279891d-d0c2-4570-a8a9-28940539bbb0","Type":"ContainerDied","Data":"fa6ac5bd9b3b00ec9c9dc61cc97f4f801bf94d5a69d919836fb29624c1d0e1ab"}
Sep 16 16:40:22.749636 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.749258    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd" event={"ID":"8279891d-d0c2-4570-a8a9-28940539bbb0","Type":"ContainerDied","Data":"268e0b5903e57ac72991ac9f77643ced921ca94c31024a7099cb51c275f1c307"}
Sep 16 16:40:22.749636 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.749300    2574 scope.go:117] "RemoveContainer" containerID="fa6ac5bd9b3b00ec9c9dc61cc97f4f801bf94d5a69d919836fb29624c1d0e1ab"
Sep 16 16:40:22.750689 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.750551    2574 generic.go:356] "Generic (PLEG): container finished" podID="38d9a68e-f401-47d6-acdb-f07936772cf3" containerID="7abe6ff130ebd29438560084cced89dffb548107fc176a02c623098063f8495c" exitCode=0
Sep 16 16:40:22.750689 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.750575    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd" event={"ID":"38d9a68e-f401-47d6-acdb-f07936772cf3","Type":"ContainerDied","Data":"7abe6ff130ebd29438560084cced89dffb548107fc176a02c623098063f8495c"}
Sep 16 16:40:22.750689 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.750606    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"
Sep 16 16:40:22.750689 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.750639    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd" event={"ID":"38d9a68e-f401-47d6-acdb-f07936772cf3","Type":"ContainerDied","Data":"cd4cd86d757d625b9122c68aa8e4902fa3b78004f8ece02c09a48a7f63ce174c"}
Sep 16 16:40:22.758961 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.758920    2574 scope.go:117] "RemoveContainer" containerID="fa6ac5bd9b3b00ec9c9dc61cc97f4f801bf94d5a69d919836fb29624c1d0e1ab"
Sep 16 16:40:22.759207 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:40:22.759188    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6ac5bd9b3b00ec9c9dc61cc97f4f801bf94d5a69d919836fb29624c1d0e1ab\": container with ID starting with fa6ac5bd9b3b00ec9c9dc61cc97f4f801bf94d5a69d919836fb29624c1d0e1ab not found: ID does not exist" containerID="fa6ac5bd9b3b00ec9c9dc61cc97f4f801bf94d5a69d919836fb29624c1d0e1ab"
Sep 16 16:40:22.759282 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.759215    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6ac5bd9b3b00ec9c9dc61cc97f4f801bf94d5a69d919836fb29624c1d0e1ab"} err="failed to get container status \"fa6ac5bd9b3b00ec9c9dc61cc97f4f801bf94d5a69d919836fb29624c1d0e1ab\": rpc error: code = NotFound desc = could not find container \"fa6ac5bd9b3b00ec9c9dc61cc97f4f801bf94d5a69d919836fb29624c1d0e1ab\": container with ID starting with fa6ac5bd9b3b00ec9c9dc61cc97f4f801bf94d5a69d919836fb29624c1d0e1ab not found: ID does not exist"
Sep 16 16:40:22.759282 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.759230    2574 scope.go:117] "RemoveContainer" containerID="7abe6ff130ebd29438560084cced89dffb548107fc176a02c623098063f8495c"
Sep 16 16:40:22.766685 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.766663    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"]
Sep 16 16:40:22.766945 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.766933    2574 scope.go:117] "RemoveContainer" containerID="7abe6ff130ebd29438560084cced89dffb548107fc176a02c623098063f8495c"
Sep 16 16:40:22.767193 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:40:22.767176    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abe6ff130ebd29438560084cced89dffb548107fc176a02c623098063f8495c\": container with ID starting with 7abe6ff130ebd29438560084cced89dffb548107fc176a02c623098063f8495c not found: ID does not exist" containerID="7abe6ff130ebd29438560084cced89dffb548107fc176a02c623098063f8495c"
Sep 16 16:40:22.767240 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.767199    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abe6ff130ebd29438560084cced89dffb548107fc176a02c623098063f8495c"} err="failed to get container status \"7abe6ff130ebd29438560084cced89dffb548107fc176a02c623098063f8495c\": rpc error: code = NotFound desc = could not find container \"7abe6ff130ebd29438560084cced89dffb548107fc176a02c623098063f8495c\": container with ID starting with 7abe6ff130ebd29438560084cced89dffb548107fc176a02c623098063f8495c not found: ID does not exist"
Sep 16 16:40:22.772334 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.772314    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-fshcd"]
Sep 16 16:40:22.781215 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.781195    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"]
Sep 16 16:40:22.784690 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:22.784669    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-rbcqd"]
Sep 16 16:40:23.994165 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:23.994130    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38d9a68e-f401-47d6-acdb-f07936772cf3" path="/var/lib/kubelet/pods/38d9a68e-f401-47d6-acdb-f07936772cf3/volumes"
Sep 16 16:40:23.994722 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:40:23.994703    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8279891d-d0c2-4570-a8a9-28940539bbb0" path="/var/lib/kubelet/pods/8279891d-d0c2-4570-a8a9-28940539bbb0/volumes"
Sep 16 16:41:06.639202 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.639165    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"]
Sep 16 16:41:06.639762 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.639741    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3787a054-fb6e-4786-b8b4-d041864a111a" containerName="main"
Sep 16 16:41:06.639856 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.639764    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3787a054-fb6e-4786-b8b4-d041864a111a" containerName="main"
Sep 16 16:41:06.639856 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.639774    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38d9a68e-f401-47d6-acdb-f07936772cf3" containerName="main"
Sep 16 16:41:06.639856 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.639781    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="38d9a68e-f401-47d6-acdb-f07936772cf3" containerName="main"
Sep 16 16:41:06.639856 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.639819    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8279891d-d0c2-4570-a8a9-28940539bbb0" containerName="main"
Sep 16 16:41:06.639856 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.639854    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8279891d-d0c2-4570-a8a9-28940539bbb0" containerName="main"
Sep 16 16:41:06.640101 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.639958    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="38d9a68e-f401-47d6-acdb-f07936772cf3" containerName="main"
Sep 16 16:41:06.640101 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.639971    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3787a054-fb6e-4786-b8b4-d041864a111a" containerName="main"
Sep 16 16:41:06.640101 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.639981    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8279891d-d0c2-4570-a8a9-28940539bbb0" containerName="main"
Sep 16 16:41:06.645017 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.644992    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"
Sep 16 16:41:06.648264 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.648239    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 16:41:06.648373 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.648309    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-t749t\""
Sep 16 16:41:06.648373 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.648322    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 16:41:06.648373 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.648356    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 16:41:06.648555 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.648486    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-cckf2\""
Sep 16 16:41:06.659138 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.659119    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"]
Sep 16 16:41:06.777555 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.777502    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38188334-ac26-4221-9cf5-c104f8875fed-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn\" (UID: \"38188334-ac26-4221-9cf5-c104f8875fed\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"
Sep 16 16:41:06.777555 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.777558    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlxtv\" (UniqueName: \"kubernetes.io/projected/38188334-ac26-4221-9cf5-c104f8875fed-kube-api-access-xlxtv\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn\" (UID: \"38188334-ac26-4221-9cf5-c104f8875fed\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"
Sep 16 16:41:06.878859 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.878823    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38188334-ac26-4221-9cf5-c104f8875fed-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn\" (UID: \"38188334-ac26-4221-9cf5-c104f8875fed\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"
Sep 16 16:41:06.879020 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.878863    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlxtv\" (UniqueName: \"kubernetes.io/projected/38188334-ac26-4221-9cf5-c104f8875fed-kube-api-access-xlxtv\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn\" (UID: \"38188334-ac26-4221-9cf5-c104f8875fed\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"
Sep 16 16:41:06.881227 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.881197    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38188334-ac26-4221-9cf5-c104f8875fed-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn\" (UID: \"38188334-ac26-4221-9cf5-c104f8875fed\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"
Sep 16 16:41:06.889580 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.889530    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlxtv\" (UniqueName: \"kubernetes.io/projected/38188334-ac26-4221-9cf5-c104f8875fed-kube-api-access-xlxtv\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn\" (UID: \"38188334-ac26-4221-9cf5-c104f8875fed\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"
Sep 16 16:41:06.956580 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:06.956551    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"
Sep 16 16:41:07.226581 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:07.226556    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"]
Sep 16 16:41:07.228383 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:41:07.228356    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38188334_ac26_4221_9cf5_c104f8875fed.slice/crio-ab5a09af423961c50e108b2f2fe7ec576fabe293fb16646c2352fbcd0c86e7a1 WatchSource:0}: Error finding container ab5a09af423961c50e108b2f2fe7ec576fabe293fb16646c2352fbcd0c86e7a1: Status 404 returned error can't find the container with id ab5a09af423961c50e108b2f2fe7ec576fabe293fb16646c2352fbcd0c86e7a1
Sep 16 16:41:07.970486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:07.970447    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn" event={"ID":"38188334-ac26-4221-9cf5-c104f8875fed","Type":"ContainerStarted","Data":"34ae5e65b4369ae648e83a43232fcd721476ed8f2cf1ab5249e5b209727c51de"}
Sep 16 16:41:07.970950 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:07.970496    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn" event={"ID":"38188334-ac26-4221-9cf5-c104f8875fed","Type":"ContainerStarted","Data":"ab5a09af423961c50e108b2f2fe7ec576fabe293fb16646c2352fbcd0c86e7a1"}
Sep 16 16:41:07.970950 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:07.970648    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"
Sep 16 16:41:07.992902 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:07.992843    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn" podStartSLOduration=1.992822595 podStartE2EDuration="1.992822595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:41:07.99023173 +0000 UTC m=+3064.509854546" watchObservedRunningTime="2026-09-16 16:41:07.992822595 +0000 UTC m=+3064.512445372"
Sep 16 16:41:38.978079 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:41:38.978050    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"
Sep 16 16:43:11.251195 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:11.251159    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"]
Sep 16 16:43:18.977764 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:18.977718    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn" podUID="38188334-ac26-4221-9cf5-c104f8875fed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:43:26.270965 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:26.270925    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn" podUID="38188334-ac26-4221-9cf5-c104f8875fed" containerName="main" containerID="cri-o://34ae5e65b4369ae648e83a43232fcd721476ed8f2cf1ab5249e5b209727c51de" gracePeriod=45
Sep 16 16:43:28.978568 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:28.978532    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn" podUID="38188334-ac26-4221-9cf5-c104f8875fed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:43:38.978180 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:38.978141    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn" podUID="38188334-ac26-4221-9cf5-c104f8875fed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:43:38.978669 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:38.978338    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"
Sep 16 16:43:48.977776 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:48.977739    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn" podUID="38188334-ac26-4221-9cf5-c104f8875fed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:43:56.512485 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.512461    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"
Sep 16 16:43:56.562883 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.562829    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38188334-ac26-4221-9cf5-c104f8875fed-tls-certs\") pod \"38188334-ac26-4221-9cf5-c104f8875fed\" (UID: \"38188334-ac26-4221-9cf5-c104f8875fed\") "
Sep 16 16:43:56.562883 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.562871    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlxtv\" (UniqueName: \"kubernetes.io/projected/38188334-ac26-4221-9cf5-c104f8875fed-kube-api-access-xlxtv\") pod \"38188334-ac26-4221-9cf5-c104f8875fed\" (UID: \"38188334-ac26-4221-9cf5-c104f8875fed\") "
Sep 16 16:43:56.564869 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.564841    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38188334-ac26-4221-9cf5-c104f8875fed-kube-api-access-xlxtv" (OuterVolumeSpecName: "kube-api-access-xlxtv") pod "38188334-ac26-4221-9cf5-c104f8875fed" (UID: "38188334-ac26-4221-9cf5-c104f8875fed"). InnerVolumeSpecName "kube-api-access-xlxtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:43:56.564968 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.564926    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38188334-ac26-4221-9cf5-c104f8875fed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38188334-ac26-4221-9cf5-c104f8875fed" (UID: "38188334-ac26-4221-9cf5-c104f8875fed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:43:56.663729 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.663702    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38188334-ac26-4221-9cf5-c104f8875fed-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:43:56.663729 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.663728    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlxtv\" (UniqueName: \"kubernetes.io/projected/38188334-ac26-4221-9cf5-c104f8875fed-kube-api-access-xlxtv\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:43:56.748505 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.748475    2574 generic.go:356] "Generic (PLEG): container finished" podID="38188334-ac26-4221-9cf5-c104f8875fed" containerID="34ae5e65b4369ae648e83a43232fcd721476ed8f2cf1ab5249e5b209727c51de" exitCode=0
Sep 16 16:43:56.748634 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.748538    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"
Sep 16 16:43:56.748634 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.748558    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn" event={"ID":"38188334-ac26-4221-9cf5-c104f8875fed","Type":"ContainerDied","Data":"34ae5e65b4369ae648e83a43232fcd721476ed8f2cf1ab5249e5b209727c51de"}
Sep 16 16:43:56.748634 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.748612    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn" event={"ID":"38188334-ac26-4221-9cf5-c104f8875fed","Type":"ContainerDied","Data":"ab5a09af423961c50e108b2f2fe7ec576fabe293fb16646c2352fbcd0c86e7a1"}
Sep 16 16:43:56.748742 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.748637    2574 scope.go:117] "RemoveContainer" containerID="34ae5e65b4369ae648e83a43232fcd721476ed8f2cf1ab5249e5b209727c51de"
Sep 16 16:43:56.757452 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.757435    2574 scope.go:117] "RemoveContainer" containerID="34ae5e65b4369ae648e83a43232fcd721476ed8f2cf1ab5249e5b209727c51de"
Sep 16 16:43:56.757751 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:43:56.757733    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ae5e65b4369ae648e83a43232fcd721476ed8f2cf1ab5249e5b209727c51de\": container with ID starting with 34ae5e65b4369ae648e83a43232fcd721476ed8f2cf1ab5249e5b209727c51de not found: ID does not exist" containerID="34ae5e65b4369ae648e83a43232fcd721476ed8f2cf1ab5249e5b209727c51de"
Sep 16 16:43:56.757816 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.757760    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ae5e65b4369ae648e83a43232fcd721476ed8f2cf1ab5249e5b209727c51de"} err="failed to get container status \"34ae5e65b4369ae648e83a43232fcd721476ed8f2cf1ab5249e5b209727c51de\": rpc error: code = NotFound desc = could not find container \"34ae5e65b4369ae648e83a43232fcd721476ed8f2cf1ab5249e5b209727c51de\": container with ID starting with 34ae5e65b4369ae648e83a43232fcd721476ed8f2cf1ab5249e5b209727c51de not found: ID does not exist"
Sep 16 16:43:56.768650 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.768625    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"]
Sep 16 16:43:56.772384 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:56.772362    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheltnfn"]
Sep 16 16:43:57.994194 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:43:57.994163    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38188334-ac26-4221-9cf5-c104f8875fed" path="/var/lib/kubelet/pods/38188334-ac26-4221-9cf5-c104f8875fed/volumes"
Sep 16 16:44:01.065290 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.065255    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"]
Sep 16 16:44:01.065682 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.065665    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38188334-ac26-4221-9cf5-c104f8875fed" containerName="main"
Sep 16 16:44:01.065682 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.065683    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="38188334-ac26-4221-9cf5-c104f8875fed" containerName="main"
Sep 16 16:44:01.065765 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.065755    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="38188334-ac26-4221-9cf5-c104f8875fed" containerName="main"
Sep 16 16:44:01.070382 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.070362    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.072763 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.072740    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 16 16:44:01.072885 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.072865    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 16:44:01.072942 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.072893    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-4mjml\""
Sep 16 16:44:01.072989 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.072966    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 16:44:01.073158 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.073140    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:44:01.073239 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.073193    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 16:44:01.081602 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.081576    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"]
Sep 16 16:44:01.199256 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.199226    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.199407 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.199263    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.199407 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.199336    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.199481 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.199402    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7740801-5312-4a99-a7fb-158cb2f88b5f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.199481 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.199453    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7ldl\" (UniqueName: \"kubernetes.io/projected/d7740801-5312-4a99-a7fb-158cb2f88b5f-kube-api-access-r7ldl\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.199481 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.199471    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.199625 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.199493    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.199625 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.199544    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7740801-5312-4a99-a7fb-158cb2f88b5f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.300091 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.300058    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.300271 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.300098    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7740801-5312-4a99-a7fb-158cb2f88b5f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.300271 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.300126    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7ldl\" (UniqueName: \"kubernetes.io/projected/d7740801-5312-4a99-a7fb-158cb2f88b5f-kube-api-access-r7ldl\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.300271 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.300153    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.300271 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.300181    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.300271 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.300210    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7740801-5312-4a99-a7fb-158cb2f88b5f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.300271 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.300267    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.300632 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.300389    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.300632 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.300620    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.300785 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.300721    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.300954 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.300933    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.301024 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.300950    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.301024 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.300967    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7740801-5312-4a99-a7fb-158cb2f88b5f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.302489 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.302467    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.302700 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.302682    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7740801-5312-4a99-a7fb-158cb2f88b5f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.311580 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.311557    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7ldl\" (UniqueName: \"kubernetes.io/projected/d7740801-5312-4a99-a7fb-158cb2f88b5f-kube-api-access-r7ldl\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.328848 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.328783    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"]
Sep 16 16:44:01.333759 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.333742    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"
Sep 16 16:44:01.335976 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.335957    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-dn8kx\""
Sep 16 16:44:01.340342 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.340317    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"]
Sep 16 16:44:01.383121 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.383097    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:01.401231 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.401208    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8413a2c8-c57b-438f-8730-79ae5c60e203-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d\" (UID: \"8413a2c8-c57b-438f-8730-79ae5c60e203\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"
Sep 16 16:44:01.401324 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.401245    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws98z\" (UniqueName: \"kubernetes.io/projected/8413a2c8-c57b-438f-8730-79ae5c60e203-kube-api-access-ws98z\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d\" (UID: \"8413a2c8-c57b-438f-8730-79ae5c60e203\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"
Sep 16 16:44:01.502307 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.502266    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8413a2c8-c57b-438f-8730-79ae5c60e203-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d\" (UID: \"8413a2c8-c57b-438f-8730-79ae5c60e203\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"
Sep 16 16:44:01.502468 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.502314    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws98z\" (UniqueName: \"kubernetes.io/projected/8413a2c8-c57b-438f-8730-79ae5c60e203-kube-api-access-ws98z\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d\" (UID: \"8413a2c8-c57b-438f-8730-79ae5c60e203\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"
Sep 16 16:44:01.504927 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.504903    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8413a2c8-c57b-438f-8730-79ae5c60e203-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d\" (UID: \"8413a2c8-c57b-438f-8730-79ae5c60e203\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"
Sep 16 16:44:01.518060 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.518037    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws98z\" (UniqueName: \"kubernetes.io/projected/8413a2c8-c57b-438f-8730-79ae5c60e203-kube-api-access-ws98z\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d\" (UID: \"8413a2c8-c57b-438f-8730-79ae5c60e203\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"
Sep 16 16:44:01.645715 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.645687    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"
Sep 16 16:44:01.661396 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.661371    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"]
Sep 16 16:44:01.663254 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:44:01.663230    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7740801_5312_4a99_a7fb_158cb2f88b5f.slice/crio-bfbaba8259d5f23d660afa982f0a699ed5ba8108d4aa8eb2e6a89e1af996fd5d WatchSource:0}: Error finding container bfbaba8259d5f23d660afa982f0a699ed5ba8108d4aa8eb2e6a89e1af996fd5d: Status 404 returned error can't find the container with id bfbaba8259d5f23d660afa982f0a699ed5ba8108d4aa8eb2e6a89e1af996fd5d
Sep 16 16:44:01.775292 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.775232    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" event={"ID":"d7740801-5312-4a99-a7fb-158cb2f88b5f","Type":"ContainerStarted","Data":"f1bda04116d9310af5116a5824029102cbe5e30da3c078e02614419a8216fc03"}
Sep 16 16:44:01.775453 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.775305    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" event={"ID":"d7740801-5312-4a99-a7fb-158cb2f88b5f","Type":"ContainerStarted","Data":"bfbaba8259d5f23d660afa982f0a699ed5ba8108d4aa8eb2e6a89e1af996fd5d"}
Sep 16 16:44:01.922179 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:01.922151    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"]
Sep 16 16:44:01.924097 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:44:01.924067    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8413a2c8_c57b_438f_8730_79ae5c60e203.slice/crio-3b126e1509daaf3e6579dd0d838965941b946dcd48bf5f5a403822f5aff8438a WatchSource:0}: Error finding container 3b126e1509daaf3e6579dd0d838965941b946dcd48bf5f5a403822f5aff8438a: Status 404 returned error can't find the container with id 3b126e1509daaf3e6579dd0d838965941b946dcd48bf5f5a403822f5aff8438a
Sep 16 16:44:02.782376 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:02.782335    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d" event={"ID":"8413a2c8-c57b-438f-8730-79ae5c60e203","Type":"ContainerStarted","Data":"c8ae7eb724b31bf897984a5f36400e22e21cdbb2e5ae9b2eed62e407d942d08e"}
Sep 16 16:44:02.782777 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:02.782401    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d" event={"ID":"8413a2c8-c57b-438f-8730-79ae5c60e203","Type":"ContainerStarted","Data":"3b126e1509daaf3e6579dd0d838965941b946dcd48bf5f5a403822f5aff8438a"}
Sep 16 16:44:02.782777 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:02.782430    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"
Sep 16 16:44:02.798869 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:02.798820    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d" podStartSLOduration=1.798805056 podStartE2EDuration="1.798805056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:44:02.797591498 +0000 UTC m=+3239.317214274" watchObservedRunningTime="2026-09-16 16:44:02.798805056 +0000 UTC m=+3239.318427831"
Sep 16 16:44:04.793554 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:04.793504    2574 generic.go:356] "Generic (PLEG): container finished" podID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerID="f1bda04116d9310af5116a5824029102cbe5e30da3c078e02614419a8216fc03" exitCode=0
Sep 16 16:44:04.794002 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:04.793573    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" event={"ID":"d7740801-5312-4a99-a7fb-158cb2f88b5f","Type":"ContainerDied","Data":"f1bda04116d9310af5116a5824029102cbe5e30da3c078e02614419a8216fc03"}
Sep 16 16:44:05.800643 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:05.800595    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" event={"ID":"d7740801-5312-4a99-a7fb-158cb2f88b5f","Type":"ContainerStarted","Data":"d429747c478a5c5a78941fac97f562a116102df04a8df30ffadae5724254363f"}
Sep 16 16:44:06.384073 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:06.384043    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:11.383584 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:11.383544    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:44:11.385415 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:11.385378    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 16:44:21.383785 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:21.383744    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 16:44:31.383702 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:31.383659    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 16:44:33.790059 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:33.790024    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"
Sep 16 16:44:33.815828 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:33.815768    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" podStartSLOduration=30.30965083 podStartE2EDuration="32.815747059s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.506096229s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:44:05.846441018 +0000 UTC m=+3242.366063793" watchObservedRunningTime="2026-09-16 16:44:33.815747059 +0000 UTC m=+3270.335369838"
Sep 16 16:44:41.384040 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:41.383993    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 16:44:51.384394 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:44:51.384350    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 16:45:01.384011 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:45:01.383956    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 16:45:11.383736 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:45:11.383691    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 16:45:21.384235 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:45:21.384187    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 16:45:31.384143 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:45:31.384044    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 16:45:41.384221 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:45:41.384170    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 16:45:51.384160 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:45:51.384117    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 16:46:01.394203 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:01.394171    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:46:01.402291 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:01.402271    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:46:05.950955 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:05.950919    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"]
Sep 16 16:46:05.957977 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:05.957946    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"]
Sep 16 16:46:13.789074 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:13.789036    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d" podUID="8413a2c8-c57b-438f-8730-79ae5c60e203" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:46:20.972129 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:20.972072    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d" podUID="8413a2c8-c57b-438f-8730-79ae5c60e203" containerName="main" containerID="cri-o://c8ae7eb724b31bf897984a5f36400e22e21cdbb2e5ae9b2eed62e407d942d08e" gracePeriod=45
Sep 16 16:46:20.981374 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:20.981331    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main" containerID="cri-o://d429747c478a5c5a78941fac97f562a116102df04a8df30ffadae5724254363f" gracePeriod=45
Sep 16 16:46:21.395668 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.395626    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 16 16:46:21.724914 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.724887    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:46:21.827010 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.826979    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7740801-5312-4a99-a7fb-158cb2f88b5f-tls-certs\") pod \"d7740801-5312-4a99-a7fb-158cb2f88b5f\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") "
Sep 16 16:46:21.827151 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.827019    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7ldl\" (UniqueName: \"kubernetes.io/projected/d7740801-5312-4a99-a7fb-158cb2f88b5f-kube-api-access-r7ldl\") pod \"d7740801-5312-4a99-a7fb-158cb2f88b5f\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") "
Sep 16 16:46:21.827151 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.827045    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-home\") pod \"d7740801-5312-4a99-a7fb-158cb2f88b5f\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") "
Sep 16 16:46:21.827151 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.827082    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7740801-5312-4a99-a7fb-158cb2f88b5f-cabundle-cert\") pod \"d7740801-5312-4a99-a7fb-158cb2f88b5f\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") "
Sep 16 16:46:21.827151 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.827131    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-dshm\") pod \"d7740801-5312-4a99-a7fb-158cb2f88b5f\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") "
Sep 16 16:46:21.827412 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.827159    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-tmp-dir\") pod \"d7740801-5312-4a99-a7fb-158cb2f88b5f\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") "
Sep 16 16:46:21.827412 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.827202    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-model-cache\") pod \"d7740801-5312-4a99-a7fb-158cb2f88b5f\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") "
Sep 16 16:46:21.827412 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.827234    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-kserve-provision-location\") pod \"d7740801-5312-4a99-a7fb-158cb2f88b5f\" (UID: \"d7740801-5312-4a99-a7fb-158cb2f88b5f\") "
Sep 16 16:46:21.827585 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.827435    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7740801-5312-4a99-a7fb-158cb2f88b5f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d7740801-5312-4a99-a7fb-158cb2f88b5f" (UID: "d7740801-5312-4a99-a7fb-158cb2f88b5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:46:21.827636 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.827572    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d7740801-5312-4a99-a7fb-158cb2f88b5f" (UID: "d7740801-5312-4a99-a7fb-158cb2f88b5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:46:21.827815 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.827794    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-home" (OuterVolumeSpecName: "home") pod "d7740801-5312-4a99-a7fb-158cb2f88b5f" (UID: "d7740801-5312-4a99-a7fb-158cb2f88b5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:46:21.830433 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.830408    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-dshm" (OuterVolumeSpecName: "dshm") pod "d7740801-5312-4a99-a7fb-158cb2f88b5f" (UID: "d7740801-5312-4a99-a7fb-158cb2f88b5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:46:21.830574 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.830425    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7740801-5312-4a99-a7fb-158cb2f88b5f-kube-api-access-r7ldl" (OuterVolumeSpecName: "kube-api-access-r7ldl") pod "d7740801-5312-4a99-a7fb-158cb2f88b5f" (UID: "d7740801-5312-4a99-a7fb-158cb2f88b5f"). InnerVolumeSpecName "kube-api-access-r7ldl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:46:21.830574 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.830411    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7740801-5312-4a99-a7fb-158cb2f88b5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d7740801-5312-4a99-a7fb-158cb2f88b5f" (UID: "d7740801-5312-4a99-a7fb-158cb2f88b5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:46:21.840606 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.840577    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d7740801-5312-4a99-a7fb-158cb2f88b5f" (UID: "d7740801-5312-4a99-a7fb-158cb2f88b5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:46:21.850112 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.850085    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d7740801-5312-4a99-a7fb-158cb2f88b5f" (UID: "d7740801-5312-4a99-a7fb-158cb2f88b5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:46:21.928072 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.928052    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7740801-5312-4a99-a7fb-158cb2f88b5f-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:46:21.928168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.928073    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7ldl\" (UniqueName: \"kubernetes.io/projected/d7740801-5312-4a99-a7fb-158cb2f88b5f-kube-api-access-r7ldl\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:46:21.928168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.928083    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:46:21.928168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.928092    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7740801-5312-4a99-a7fb-158cb2f88b5f-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:46:21.928168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.928100    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:46:21.928168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.928107    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:46:21.928168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.928114    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:46:21.928168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:21.928122    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7740801-5312-4a99-a7fb-158cb2f88b5f-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:46:22.477766 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:22.477730    2574 generic.go:356] "Generic (PLEG): container finished" podID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerID="d429747c478a5c5a78941fac97f562a116102df04a8df30ffadae5724254363f" exitCode=0
Sep 16 16:46:22.478132 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:22.477788    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"
Sep 16 16:46:22.478132 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:22.477814    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" event={"ID":"d7740801-5312-4a99-a7fb-158cb2f88b5f","Type":"ContainerDied","Data":"d429747c478a5c5a78941fac97f562a116102df04a8df30ffadae5724254363f"}
Sep 16 16:46:22.478132 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:22.477873    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5" event={"ID":"d7740801-5312-4a99-a7fb-158cb2f88b5f","Type":"ContainerDied","Data":"bfbaba8259d5f23d660afa982f0a699ed5ba8108d4aa8eb2e6a89e1af996fd5d"}
Sep 16 16:46:22.478132 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:22.477904    2574 scope.go:117] "RemoveContainer" containerID="d429747c478a5c5a78941fac97f562a116102df04a8df30ffadae5724254363f"
Sep 16 16:46:22.486278 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:22.486258    2574 scope.go:117] "RemoveContainer" containerID="f1bda04116d9310af5116a5824029102cbe5e30da3c078e02614419a8216fc03"
Sep 16 16:46:22.493677 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:22.493654    2574 scope.go:117] "RemoveContainer" containerID="d429747c478a5c5a78941fac97f562a116102df04a8df30ffadae5724254363f"
Sep 16 16:46:22.493979 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:46:22.493961    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d429747c478a5c5a78941fac97f562a116102df04a8df30ffadae5724254363f\": container with ID starting with d429747c478a5c5a78941fac97f562a116102df04a8df30ffadae5724254363f not found: ID does not exist" containerID="d429747c478a5c5a78941fac97f562a116102df04a8df30ffadae5724254363f"
Sep 16 16:46:22.494042 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:22.493986    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d429747c478a5c5a78941fac97f562a116102df04a8df30ffadae5724254363f"} err="failed to get container status \"d429747c478a5c5a78941fac97f562a116102df04a8df30ffadae5724254363f\": rpc error: code = NotFound desc = could not find container \"d429747c478a5c5a78941fac97f562a116102df04a8df30ffadae5724254363f\": container with ID starting with d429747c478a5c5a78941fac97f562a116102df04a8df30ffadae5724254363f not found: ID does not exist"
Sep 16 16:46:22.494042 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:22.494002    2574 scope.go:117] "RemoveContainer" containerID="f1bda04116d9310af5116a5824029102cbe5e30da3c078e02614419a8216fc03"
Sep 16 16:46:22.494329 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:46:22.494303    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1bda04116d9310af5116a5824029102cbe5e30da3c078e02614419a8216fc03\": container with ID starting with f1bda04116d9310af5116a5824029102cbe5e30da3c078e02614419a8216fc03 not found: ID does not exist" containerID="f1bda04116d9310af5116a5824029102cbe5e30da3c078e02614419a8216fc03"
Sep 16 16:46:22.494418 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:22.494336    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1bda04116d9310af5116a5824029102cbe5e30da3c078e02614419a8216fc03"} err="failed to get container status \"f1bda04116d9310af5116a5824029102cbe5e30da3c078e02614419a8216fc03\": rpc error: code = NotFound desc = could not find container \"f1bda04116d9310af5116a5824029102cbe5e30da3c078e02614419a8216fc03\": container with ID starting with f1bda04116d9310af5116a5824029102cbe5e30da3c078e02614419a8216fc03 not found: ID does not exist"
Sep 16 16:46:22.496222 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:22.496178    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"]
Sep 16 16:46:22.499377 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:22.499356    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-9b88b59f5-lbwk5"]
Sep 16 16:46:23.789178 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:23.789140    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d" podUID="8413a2c8-c57b-438f-8730-79ae5c60e203" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:46:23.994718 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:23.994689    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" path="/var/lib/kubelet/pods/d7740801-5312-4a99-a7fb-158cb2f88b5f/volumes"
Sep 16 16:46:33.789220 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:33.789182    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d" podUID="8413a2c8-c57b-438f-8730-79ae5c60e203" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:46:33.789636 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:33.789284    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"
Sep 16 16:46:43.789366 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:43.789285    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d" podUID="8413a2c8-c57b-438f-8730-79ae5c60e203" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:46:51.225486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.225459    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"
Sep 16 16:46:51.282279 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.282204    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws98z\" (UniqueName: \"kubernetes.io/projected/8413a2c8-c57b-438f-8730-79ae5c60e203-kube-api-access-ws98z\") pod \"8413a2c8-c57b-438f-8730-79ae5c60e203\" (UID: \"8413a2c8-c57b-438f-8730-79ae5c60e203\") "
Sep 16 16:46:51.282279 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.282256    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8413a2c8-c57b-438f-8730-79ae5c60e203-tls-certs\") pod \"8413a2c8-c57b-438f-8730-79ae5c60e203\" (UID: \"8413a2c8-c57b-438f-8730-79ae5c60e203\") "
Sep 16 16:46:51.284234 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.284209    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8413a2c8-c57b-438f-8730-79ae5c60e203-kube-api-access-ws98z" (OuterVolumeSpecName: "kube-api-access-ws98z") pod "8413a2c8-c57b-438f-8730-79ae5c60e203" (UID: "8413a2c8-c57b-438f-8730-79ae5c60e203"). InnerVolumeSpecName "kube-api-access-ws98z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:46:51.284326 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.284226    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8413a2c8-c57b-438f-8730-79ae5c60e203-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8413a2c8-c57b-438f-8730-79ae5c60e203" (UID: "8413a2c8-c57b-438f-8730-79ae5c60e203"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:46:51.383881 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.383850    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws98z\" (UniqueName: \"kubernetes.io/projected/8413a2c8-c57b-438f-8730-79ae5c60e203-kube-api-access-ws98z\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:46:51.383881 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.383877    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8413a2c8-c57b-438f-8730-79ae5c60e203-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:46:51.617118 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.617083    2574 generic.go:356] "Generic (PLEG): container finished" podID="8413a2c8-c57b-438f-8730-79ae5c60e203" containerID="c8ae7eb724b31bf897984a5f36400e22e21cdbb2e5ae9b2eed62e407d942d08e" exitCode=0
Sep 16 16:46:51.617282 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.617128    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"
Sep 16 16:46:51.617282 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.617133    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d" event={"ID":"8413a2c8-c57b-438f-8730-79ae5c60e203","Type":"ContainerDied","Data":"c8ae7eb724b31bf897984a5f36400e22e21cdbb2e5ae9b2eed62e407d942d08e"}
Sep 16 16:46:51.617282 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.617256    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d" event={"ID":"8413a2c8-c57b-438f-8730-79ae5c60e203","Type":"ContainerDied","Data":"3b126e1509daaf3e6579dd0d838965941b946dcd48bf5f5a403822f5aff8438a"}
Sep 16 16:46:51.617389 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.617287    2574 scope.go:117] "RemoveContainer" containerID="c8ae7eb724b31bf897984a5f36400e22e21cdbb2e5ae9b2eed62e407d942d08e"
Sep 16 16:46:51.626252 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.626234    2574 scope.go:117] "RemoveContainer" containerID="c8ae7eb724b31bf897984a5f36400e22e21cdbb2e5ae9b2eed62e407d942d08e"
Sep 16 16:46:51.626538 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:46:51.626502    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ae7eb724b31bf897984a5f36400e22e21cdbb2e5ae9b2eed62e407d942d08e\": container with ID starting with c8ae7eb724b31bf897984a5f36400e22e21cdbb2e5ae9b2eed62e407d942d08e not found: ID does not exist" containerID="c8ae7eb724b31bf897984a5f36400e22e21cdbb2e5ae9b2eed62e407d942d08e"
Sep 16 16:46:51.626601 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.626543    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ae7eb724b31bf897984a5f36400e22e21cdbb2e5ae9b2eed62e407d942d08e"} err="failed to get container status \"c8ae7eb724b31bf897984a5f36400e22e21cdbb2e5ae9b2eed62e407d942d08e\": rpc error: code = NotFound desc = could not find container \"c8ae7eb724b31bf897984a5f36400e22e21cdbb2e5ae9b2eed62e407d942d08e\": container with ID starting with c8ae7eb724b31bf897984a5f36400e22e21cdbb2e5ae9b2eed62e407d942d08e not found: ID does not exist"
Sep 16 16:46:51.639109 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.639086    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"]
Sep 16 16:46:51.643169 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.643147    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7b69d"]
Sep 16 16:46:51.994773 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:46:51.994695    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8413a2c8-c57b-438f-8730-79ae5c60e203" path="/var/lib/kubelet/pods/8413a2c8-c57b-438f-8730-79ae5c60e203/volumes"
Sep 16 16:47:07.174660 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.174624    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"]
Sep 16 16:47:07.175114 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.175017    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8413a2c8-c57b-438f-8730-79ae5c60e203" containerName="main"
Sep 16 16:47:07.175114 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.175029    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8413a2c8-c57b-438f-8730-79ae5c60e203" containerName="main"
Sep 16 16:47:07.175114 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.175044    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main"
Sep 16 16:47:07.175114 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.175049    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main"
Sep 16 16:47:07.175114 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.175061    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="storage-initializer"
Sep 16 16:47:07.175114 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.175067    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="storage-initializer"
Sep 16 16:47:07.175425 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.175122    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8413a2c8-c57b-438f-8730-79ae5c60e203" containerName="main"
Sep 16 16:47:07.175425 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.175130    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7740801-5312-4a99-a7fb-158cb2f88b5f" containerName="main"
Sep 16 16:47:07.179898 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.179870    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"
Sep 16 16:47:07.182828 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.182780    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-ms98d\""
Sep 16 16:47:07.182962 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.182780    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 16:47:07.182962 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.182891    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-l2jcm\""
Sep 16 16:47:07.182962 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.182795    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 16:47:07.183136 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.182795    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 16:47:07.188745 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.188685    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"]
Sep 16 16:47:07.321015 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.320982    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ae2670-0e1d-407d-8370-b98af2da42be-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7\" (UID: \"55ae2670-0e1d-407d-8370-b98af2da42be\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"
Sep 16 16:47:07.321178 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.321020    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkwlg\" (UniqueName: \"kubernetes.io/projected/55ae2670-0e1d-407d-8370-b98af2da42be-kube-api-access-nkwlg\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7\" (UID: \"55ae2670-0e1d-407d-8370-b98af2da42be\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"
Sep 16 16:47:07.422180 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.422149    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ae2670-0e1d-407d-8370-b98af2da42be-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7\" (UID: \"55ae2670-0e1d-407d-8370-b98af2da42be\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"
Sep 16 16:47:07.422388 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.422189    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkwlg\" (UniqueName: \"kubernetes.io/projected/55ae2670-0e1d-407d-8370-b98af2da42be-kube-api-access-nkwlg\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7\" (UID: \"55ae2670-0e1d-407d-8370-b98af2da42be\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"
Sep 16 16:47:07.424588 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.424556    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ae2670-0e1d-407d-8370-b98af2da42be-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7\" (UID: \"55ae2670-0e1d-407d-8370-b98af2da42be\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"
Sep 16 16:47:07.433937 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.433884    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkwlg\" (UniqueName: \"kubernetes.io/projected/55ae2670-0e1d-407d-8370-b98af2da42be-kube-api-access-nkwlg\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7\" (UID: \"55ae2670-0e1d-407d-8370-b98af2da42be\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"
Sep 16 16:47:07.492790 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.492770    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"
Sep 16 16:47:07.760301 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:07.760225    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"]
Sep 16 16:47:07.763289 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:47:07.763263    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ae2670_0e1d_407d_8370_b98af2da42be.slice/crio-27627e6f9d5c75cb9e5ad6db1b996cba86a9601ebb925cad123777efe64e4b3a WatchSource:0}: Error finding container 27627e6f9d5c75cb9e5ad6db1b996cba86a9601ebb925cad123777efe64e4b3a: Status 404 returned error can't find the container with id 27627e6f9d5c75cb9e5ad6db1b996cba86a9601ebb925cad123777efe64e4b3a
Sep 16 16:47:08.698956 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:08.698923    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7" event={"ID":"55ae2670-0e1d-407d-8370-b98af2da42be","Type":"ContainerStarted","Data":"72730007cc2c0791a1399866d13845860a037b6ff225f6fc9560af3739d85263"}
Sep 16 16:47:08.699300 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:08.698971    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7" event={"ID":"55ae2670-0e1d-407d-8370-b98af2da42be","Type":"ContainerStarted","Data":"27627e6f9d5c75cb9e5ad6db1b996cba86a9601ebb925cad123777efe64e4b3a"}
Sep 16 16:47:08.699300 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:08.699066    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"
Sep 16 16:47:08.715720 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:08.715659    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7" podStartSLOduration=1.715641169 podStartE2EDuration="1.715641169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:47:08.71473221 +0000 UTC m=+3425.234354987" watchObservedRunningTime="2026-09-16 16:47:08.715641169 +0000 UTC m=+3425.235263946"
Sep 16 16:47:20.666760 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.666724    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"]
Sep 16 16:47:20.670529 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.670491    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.672549 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.672512    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:47:20.672675 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.672650    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-wwq9r\""
Sep 16 16:47:20.673080 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.673061    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 16:47:20.673162 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.673102    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 16:47:20.673318 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.673301    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 16:47:20.673387 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.673336    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 16:47:20.681739 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.681715    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"]
Sep 16 16:47:20.735335 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.735302    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn7pr\" (UniqueName: \"kubernetes.io/projected/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-kube-api-access-hn7pr\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.735477 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.735340    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-model-cache\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.735477 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.735370    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-tls-certs\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.735477 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.735464    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-cabundle-cert\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.735642 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.735494    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-dshm\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.735642 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.735597    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-kserve-provision-location\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.735642 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.735628    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-tmp-dir\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.735770 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.735650    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-home\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.836876 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.836844    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-cabundle-cert\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.836876 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.836880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-dshm\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.837095 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.836912    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-kserve-provision-location\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.837095 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.836938    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-tmp-dir\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.837095 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.836967    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-home\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.837095 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.837016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn7pr\" (UniqueName: \"kubernetes.io/projected/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-kube-api-access-hn7pr\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.837302 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.837145    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-model-cache\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.837302 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.837192    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-tls-certs\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.837401 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.837357    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-home\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.837456 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.837393    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-kserve-provision-location\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.837514 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.837475    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-tmp-dir\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.837611 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.837586    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-model-cache\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.837819 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.837799    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-cabundle-cert\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.839107 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.839088    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-dshm\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.839673 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.839656    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-tls-certs\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.845728 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.845711    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn7pr\" (UniqueName: \"kubernetes.io/projected/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-kube-api-access-hn7pr\") pod \"stop-feature-test-kserve-dc4484968-v86zv\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:20.982664 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:20.982583    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:21.258267 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:21.258243    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"]
Sep 16 16:47:21.764631 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:21.764554    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" event={"ID":"dc03d54f-bf87-4cc1-86f2-34cbea3664ff","Type":"ContainerStarted","Data":"abbaae2b463aac0fac03c0ea9fd6ded446f8a351540c428bad612dfb0234200a"}
Sep 16 16:47:21.764631 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:21.764626    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" event={"ID":"dc03d54f-bf87-4cc1-86f2-34cbea3664ff","Type":"ContainerStarted","Data":"27986caf214e14d48e95c0fe5d2f69aeef8261f2dde0a2c28929855224ccd485"}
Sep 16 16:47:23.775439 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:23.775407    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerID="abbaae2b463aac0fac03c0ea9fd6ded446f8a351540c428bad612dfb0234200a" exitCode=0
Sep 16 16:47:23.775868 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:23.775479    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" event={"ID":"dc03d54f-bf87-4cc1-86f2-34cbea3664ff","Type":"ContainerDied","Data":"abbaae2b463aac0fac03c0ea9fd6ded446f8a351540c428bad612dfb0234200a"}
Sep 16 16:47:24.782871 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:24.782836    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" event={"ID":"dc03d54f-bf87-4cc1-86f2-34cbea3664ff","Type":"ContainerStarted","Data":"11497ff9ade0fec416b78d4e503e0221e17b07773c5fd17fd17047b0d2309923"}
Sep 16 16:47:24.803020 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:24.802958    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" podStartSLOduration=2.5766821760000003 podStartE2EDuration="4.802937661s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.226255485s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:47:24.802640625 +0000 UTC m=+3441.322263400" watchObservedRunningTime="2026-09-16 16:47:24.802937661 +0000 UTC m=+3441.322560439"
Sep 16 16:47:24.983671 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:24.983644    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:30.983730 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:30.983682    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:47:30.985293 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:30.985264    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 16:47:39.705438 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:39.705402    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"
Sep 16 16:47:40.983759 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:40.983697    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 16:47:50.983511 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:47:50.983469    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 16:48:00.983391 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:48:00.983344    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 16:48:10.983805 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:48:10.983762    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 16:48:20.983617 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:48:20.983564    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 16:48:30.983125 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:48:30.983021    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 16:48:40.983195 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:48:40.983141    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 16:48:50.983245 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:48:50.983194    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 16:49:00.983775 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:00.983724    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 16:49:10.993379 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:10.993344    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:49:11.000810 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:11.000785    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:49:12.684473 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:12.684437    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"]
Sep 16 16:49:27.706191 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:27.706154    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main" containerID="cri-o://11497ff9ade0fec416b78d4e503e0221e17b07773c5fd17fd17047b0d2309923" gracePeriod=45
Sep 16 16:49:27.994131 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:27.994035    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 16:49:28.411757 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.411725    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerID="11497ff9ade0fec416b78d4e503e0221e17b07773c5fd17fd17047b0d2309923" exitCode=0
Sep 16 16:49:28.411907 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.411783    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" event={"ID":"dc03d54f-bf87-4cc1-86f2-34cbea3664ff","Type":"ContainerDied","Data":"11497ff9ade0fec416b78d4e503e0221e17b07773c5fd17fd17047b0d2309923"}
Sep 16 16:49:28.457561 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.457534    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:49:28.621665 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.621634    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-home\") pod \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") "
Sep 16 16:49:28.621825 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.621674    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-kserve-provision-location\") pod \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") "
Sep 16 16:49:28.621825 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.621699    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-cabundle-cert\") pod \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") "
Sep 16 16:49:28.621825 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.621727    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn7pr\" (UniqueName: \"kubernetes.io/projected/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-kube-api-access-hn7pr\") pod \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") "
Sep 16 16:49:28.621825 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.621765    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-model-cache\") pod \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") "
Sep 16 16:49:28.621825 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.621790    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-tmp-dir\") pod \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") "
Sep 16 16:49:28.622081 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.621831    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-dshm\") pod \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") "
Sep 16 16:49:28.622081 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.621881    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-tls-certs\") pod \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\" (UID: \"dc03d54f-bf87-4cc1-86f2-34cbea3664ff\") "
Sep 16 16:49:28.622081 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.622020    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc03d54f-bf87-4cc1-86f2-34cbea3664ff" (UID: "dc03d54f-bf87-4cc1-86f2-34cbea3664ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:49:28.622236 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.622115    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc03d54f-bf87-4cc1-86f2-34cbea3664ff" (UID: "dc03d54f-bf87-4cc1-86f2-34cbea3664ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:49:28.622236 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.622220    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:49:28.622236 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.622232    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:49:28.622409 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.622386    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-home" (OuterVolumeSpecName: "home") pod "dc03d54f-bf87-4cc1-86f2-34cbea3664ff" (UID: "dc03d54f-bf87-4cc1-86f2-34cbea3664ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:49:28.624122 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.624094    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc03d54f-bf87-4cc1-86f2-34cbea3664ff" (UID: "dc03d54f-bf87-4cc1-86f2-34cbea3664ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:49:28.624224 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.624133    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-kube-api-access-hn7pr" (OuterVolumeSpecName: "kube-api-access-hn7pr") pod "dc03d54f-bf87-4cc1-86f2-34cbea3664ff" (UID: "dc03d54f-bf87-4cc1-86f2-34cbea3664ff"). InnerVolumeSpecName "kube-api-access-hn7pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:49:28.624224 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.624163    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-dshm" (OuterVolumeSpecName: "dshm") pod "dc03d54f-bf87-4cc1-86f2-34cbea3664ff" (UID: "dc03d54f-bf87-4cc1-86f2-34cbea3664ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:49:28.639757 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.639712    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc03d54f-bf87-4cc1-86f2-34cbea3664ff" (UID: "dc03d54f-bf87-4cc1-86f2-34cbea3664ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:49:28.650963 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.650940    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc03d54f-bf87-4cc1-86f2-34cbea3664ff" (UID: "dc03d54f-bf87-4cc1-86f2-34cbea3664ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:49:28.723190 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.723165    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:49:28.723190 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.723188    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:49:28.723495 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.723199    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn7pr\" (UniqueName: \"kubernetes.io/projected/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-kube-api-access-hn7pr\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:49:28.723495 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.723210    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:49:28.723495 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.723228    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:49:28.723495 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:28.723235    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc03d54f-bf87-4cc1-86f2-34cbea3664ff-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:49:29.418742 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:29.418713    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"
Sep 16 16:49:29.418742 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:29.418734    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv" event={"ID":"dc03d54f-bf87-4cc1-86f2-34cbea3664ff","Type":"ContainerDied","Data":"27986caf214e14d48e95c0fe5d2f69aeef8261f2dde0a2c28929855224ccd485"}
Sep 16 16:49:29.418961 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:29.418799    2574 scope.go:117] "RemoveContainer" containerID="11497ff9ade0fec416b78d4e503e0221e17b07773c5fd17fd17047b0d2309923"
Sep 16 16:49:29.428460 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:29.428441    2574 scope.go:117] "RemoveContainer" containerID="abbaae2b463aac0fac03c0ea9fd6ded446f8a351540c428bad612dfb0234200a"
Sep 16 16:49:29.440799 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:29.440761    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"]
Sep 16 16:49:29.445282 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:29.445260    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-v86zv"]
Sep 16 16:49:29.994928 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:29.994886    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" path="/var/lib/kubelet/pods/dc03d54f-bf87-4cc1-86f2-34cbea3664ff/volumes"
Sep 16 16:49:41.697947 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.697918    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"]
Sep 16 16:49:41.698330 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.698317    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main"
Sep 16 16:49:41.698374 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.698332    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main"
Sep 16 16:49:41.698374 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.698345    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="storage-initializer"
Sep 16 16:49:41.698374 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.698351    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="storage-initializer"
Sep 16 16:49:41.698466 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.698431    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc03d54f-bf87-4cc1-86f2-34cbea3664ff" containerName="main"
Sep 16 16:49:41.703276 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.703256    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.705599 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.705553    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 16:49:41.705599 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.705566    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 16:49:41.705788 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.705767    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 16:49:41.705844 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.705789    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 16:49:41.705844 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.705819    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:49:41.705967 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.705925    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-wwq9r\""
Sep 16 16:49:41.714609 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.714589    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"]
Sep 16 16:49:41.741778 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.741755    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-dshm\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.741878 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.741782    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-tmp-dir\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.741878 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.741801    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-tls-certs\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.741969 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.741888    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-home\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.741969 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.741905    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w4sh\" (UniqueName: \"kubernetes.io/projected/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-kube-api-access-2w4sh\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.741969 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.741927    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-model-cache\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.741969 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.741952    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-kserve-provision-location\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.742106 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.741977    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-cabundle-cert\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.843394 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.843365    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-cabundle-cert\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.843584 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.843433    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-dshm\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.843584 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.843454    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-tmp-dir\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.843584 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.843471    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-tls-certs\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.843584 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.843495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-home\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.843584 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.843510    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w4sh\" (UniqueName: \"kubernetes.io/projected/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-kube-api-access-2w4sh\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.843871 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.843639    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-model-cache\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.843871 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.843684    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-kserve-provision-location\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.843976 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.843897    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-home\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.844038 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.843963    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-tmp-dir\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.844087 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.844044    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-kserve-provision-location\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.844157 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.844137    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-cabundle-cert\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.844212 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.844162    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-model-cache\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.845845 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.845826    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-dshm\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.846048 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.846033    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-tls-certs\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:41.851028 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:41.851004    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w4sh\" (UniqueName: \"kubernetes.io/projected/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-kube-api-access-2w4sh\") pod \"stop-feature-test-kserve-dc4484968-sf7rg\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:42.014869 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:42.014775    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:42.291198 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:42.291120    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"]
Sep 16 16:49:42.294357 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:49:42.294328    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6f669b_eef8_4b22_a5ed_46c7cf4671a6.slice/crio-8f38b65fc7800808c55133a83ef89996cd4028dff21d12da02faf41ea3e103fc WatchSource:0}: Error finding container 8f38b65fc7800808c55133a83ef89996cd4028dff21d12da02faf41ea3e103fc: Status 404 returned error can't find the container with id 8f38b65fc7800808c55133a83ef89996cd4028dff21d12da02faf41ea3e103fc
Sep 16 16:49:42.481030 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:42.480983    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" event={"ID":"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6","Type":"ContainerStarted","Data":"1398dae09afb8fc8ad40975f177e4940b64ee5760e8d9798ccb4bcc393080984"}
Sep 16 16:49:42.481221 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:42.481041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" event={"ID":"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6","Type":"ContainerStarted","Data":"8f38b65fc7800808c55133a83ef89996cd4028dff21d12da02faf41ea3e103fc"}
Sep 16 16:49:44.791217 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:44.791125    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"]
Sep 16 16:49:45.497982 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:45.497944    2574 generic.go:356] "Generic (PLEG): container finished" podID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerID="1398dae09afb8fc8ad40975f177e4940b64ee5760e8d9798ccb4bcc393080984" exitCode=0
Sep 16 16:49:45.498180 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:45.498017    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" event={"ID":"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6","Type":"ContainerDied","Data":"1398dae09afb8fc8ad40975f177e4940b64ee5760e8d9798ccb4bcc393080984"}
Sep 16 16:49:46.505361 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:46.505326    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" event={"ID":"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6","Type":"ContainerStarted","Data":"875b383d77888c90cce95bf5524f88930fbca9f3d227df70a2de8addbffb6355"}
Sep 16 16:49:46.529933 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:46.529869    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" podStartSLOduration=3.365799525 podStartE2EDuration="5.52984916s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.164049635s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:49:46.52647883 +0000 UTC m=+3583.046101597" watchObservedRunningTime="2026-09-16 16:49:46.52984916 +0000 UTC m=+3583.049471937"
Sep 16 16:49:47.015786 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:47.015745    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:49.706257 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:49.706152    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7" podUID="55ae2670-0e1d-407d-8370-b98af2da42be" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:49:52.015200 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:52.015161    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:49:52.017005 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:52.016967    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 16:49:59.706085 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:59.706000    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7" podUID="55ae2670-0e1d-407d-8370-b98af2da42be" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:49:59.810784 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:49:59.810736    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7" podUID="55ae2670-0e1d-407d-8370-b98af2da42be" containerName="main" containerID="cri-o://72730007cc2c0791a1399866d13845860a037b6ff225f6fc9560af3739d85263" gracePeriod=45
Sep 16 16:50:02.015168 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:02.015125    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 16:50:09.706096 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:09.706056    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7" podUID="55ae2670-0e1d-407d-8370-b98af2da42be" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:50:09.706692 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:09.706158    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"
Sep 16 16:50:12.015904 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:12.015855    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 16:50:19.713462 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:19.713418    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7" podUID="55ae2670-0e1d-407d-8370-b98af2da42be" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:50:22.015379 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:22.015340    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 16:50:29.705725 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:29.705684    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7" podUID="55ae2670-0e1d-407d-8370-b98af2da42be" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:50:30.060097 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.060075    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"
Sep 16 16:50:30.216695 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.216660    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkwlg\" (UniqueName: \"kubernetes.io/projected/55ae2670-0e1d-407d-8370-b98af2da42be-kube-api-access-nkwlg\") pod \"55ae2670-0e1d-407d-8370-b98af2da42be\" (UID: \"55ae2670-0e1d-407d-8370-b98af2da42be\") "
Sep 16 16:50:30.216858 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.216795    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ae2670-0e1d-407d-8370-b98af2da42be-tls-certs\") pod \"55ae2670-0e1d-407d-8370-b98af2da42be\" (UID: \"55ae2670-0e1d-407d-8370-b98af2da42be\") "
Sep 16 16:50:30.218943 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.218902    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ae2670-0e1d-407d-8370-b98af2da42be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55ae2670-0e1d-407d-8370-b98af2da42be" (UID: "55ae2670-0e1d-407d-8370-b98af2da42be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:50:30.218943 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.218919    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ae2670-0e1d-407d-8370-b98af2da42be-kube-api-access-nkwlg" (OuterVolumeSpecName: "kube-api-access-nkwlg") pod "55ae2670-0e1d-407d-8370-b98af2da42be" (UID: "55ae2670-0e1d-407d-8370-b98af2da42be"). InnerVolumeSpecName "kube-api-access-nkwlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:50:30.318123 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.318041    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ae2670-0e1d-407d-8370-b98af2da42be-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:50:30.318123 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.318071    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkwlg\" (UniqueName: \"kubernetes.io/projected/55ae2670-0e1d-407d-8370-b98af2da42be-kube-api-access-nkwlg\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:50:30.737745 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.737707    2574 generic.go:356] "Generic (PLEG): container finished" podID="55ae2670-0e1d-407d-8370-b98af2da42be" containerID="72730007cc2c0791a1399866d13845860a037b6ff225f6fc9560af3739d85263" exitCode=0
Sep 16 16:50:30.738227 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.737771    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"
Sep 16 16:50:30.738227 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.737792    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7" event={"ID":"55ae2670-0e1d-407d-8370-b98af2da42be","Type":"ContainerDied","Data":"72730007cc2c0791a1399866d13845860a037b6ff225f6fc9560af3739d85263"}
Sep 16 16:50:30.738227 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.737849    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7" event={"ID":"55ae2670-0e1d-407d-8370-b98af2da42be","Type":"ContainerDied","Data":"27627e6f9d5c75cb9e5ad6db1b996cba86a9601ebb925cad123777efe64e4b3a"}
Sep 16 16:50:30.738227 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.737881    2574 scope.go:117] "RemoveContainer" containerID="72730007cc2c0791a1399866d13845860a037b6ff225f6fc9560af3739d85263"
Sep 16 16:50:30.746664 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.746637    2574 scope.go:117] "RemoveContainer" containerID="72730007cc2c0791a1399866d13845860a037b6ff225f6fc9560af3739d85263"
Sep 16 16:50:30.746918 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:50:30.746899    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72730007cc2c0791a1399866d13845860a037b6ff225f6fc9560af3739d85263\": container with ID starting with 72730007cc2c0791a1399866d13845860a037b6ff225f6fc9560af3739d85263 not found: ID does not exist" containerID="72730007cc2c0791a1399866d13845860a037b6ff225f6fc9560af3739d85263"
Sep 16 16:50:30.746979 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.746925    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72730007cc2c0791a1399866d13845860a037b6ff225f6fc9560af3739d85263"} err="failed to get container status \"72730007cc2c0791a1399866d13845860a037b6ff225f6fc9560af3739d85263\": rpc error: code = NotFound desc = could not find container \"72730007cc2c0791a1399866d13845860a037b6ff225f6fc9560af3739d85263\": container with ID starting with 72730007cc2c0791a1399866d13845860a037b6ff225f6fc9560af3739d85263 not found: ID does not exist"
Sep 16 16:50:30.761319 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.761289    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"]
Sep 16 16:50:30.767423 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:30.767398    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebsrf7"]
Sep 16 16:50:31.995280 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:31.995244    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55ae2670-0e1d-407d-8370-b98af2da42be" path="/var/lib/kubelet/pods/55ae2670-0e1d-407d-8370-b98af2da42be/volumes"
Sep 16 16:50:32.015497 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:32.015461    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 16:50:35.874992 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:35.874954    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"]
Sep 16 16:50:35.875459 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:35.875432    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ae2670-0e1d-407d-8370-b98af2da42be" containerName="main"
Sep 16 16:50:35.875459 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:35.875447    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ae2670-0e1d-407d-8370-b98af2da42be" containerName="main"
Sep 16 16:50:35.875621 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:35.875502    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="55ae2670-0e1d-407d-8370-b98af2da42be" containerName="main"
Sep 16 16:50:35.880177 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:35.880152    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"
Sep 16 16:50:35.882680 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:35.882652    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 16:50:35.882794 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:35.882734    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 16:50:35.882846 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:35.882738    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-d7dsx\""
Sep 16 16:50:35.883346 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:35.883322    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-kllwv\""
Sep 16 16:50:35.883450 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:35.883322    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 16:50:35.894534 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:35.893076    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"]
Sep 16 16:50:35.975498 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:35.975455    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f6n8\" (UniqueName: \"kubernetes.io/projected/712e8481-0875-424b-94d9-e5f06444993f-kube-api-access-5f6n8\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt\" (UID: \"712e8481-0875-424b-94d9-e5f06444993f\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"
Sep 16 16:50:35.975687 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:35.975600    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712e8481-0875-424b-94d9-e5f06444993f-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt\" (UID: \"712e8481-0875-424b-94d9-e5f06444993f\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"
Sep 16 16:50:36.076569 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:36.076509    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712e8481-0875-424b-94d9-e5f06444993f-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt\" (UID: \"712e8481-0875-424b-94d9-e5f06444993f\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"
Sep 16 16:50:36.076762 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:36.076626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f6n8\" (UniqueName: \"kubernetes.io/projected/712e8481-0875-424b-94d9-e5f06444993f-kube-api-access-5f6n8\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt\" (UID: \"712e8481-0875-424b-94d9-e5f06444993f\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"
Sep 16 16:50:36.078981 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:36.078953    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712e8481-0875-424b-94d9-e5f06444993f-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt\" (UID: \"712e8481-0875-424b-94d9-e5f06444993f\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"
Sep 16 16:50:36.085013 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:36.084990    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f6n8\" (UniqueName: \"kubernetes.io/projected/712e8481-0875-424b-94d9-e5f06444993f-kube-api-access-5f6n8\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt\" (UID: \"712e8481-0875-424b-94d9-e5f06444993f\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"
Sep 16 16:50:36.191854 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:36.191762    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"
Sep 16 16:50:36.461849 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:36.461823    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"]
Sep 16 16:50:36.464838 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:50:36.464806    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712e8481_0875_424b_94d9_e5f06444993f.slice/crio-9c35a4c67d9af1ba63919aba2293ae69e60b7715d4d183be892d9e48b496898b WatchSource:0}: Error finding container 9c35a4c67d9af1ba63919aba2293ae69e60b7715d4d183be892d9e48b496898b: Status 404 returned error can't find the container with id 9c35a4c67d9af1ba63919aba2293ae69e60b7715d4d183be892d9e48b496898b
Sep 16 16:50:36.771370 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:36.771260    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt" event={"ID":"712e8481-0875-424b-94d9-e5f06444993f","Type":"ContainerStarted","Data":"3d71a543aeda3a51c220a19101b6756568521dc13e74016daa35913d135cd6e8"}
Sep 16 16:50:36.771559 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:36.771401    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"
Sep 16 16:50:36.771559 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:36.771428    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt" event={"ID":"712e8481-0875-424b-94d9-e5f06444993f","Type":"ContainerStarted","Data":"9c35a4c67d9af1ba63919aba2293ae69e60b7715d4d183be892d9e48b496898b"}
Sep 16 16:50:36.791232 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:36.791178    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt" podStartSLOduration=1.791159156 podStartE2EDuration="1.791159156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:50:36.78769075 +0000 UTC m=+3633.307313528" watchObservedRunningTime="2026-09-16 16:50:36.791159156 +0000 UTC m=+3633.310781934"
Sep 16 16:50:42.015785 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:42.015691    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 16:50:52.015421 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:50:52.015381    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 16:51:02.015804 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:02.015752    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 16:51:07.781171 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:07.781136    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"
Sep 16 16:51:12.015869 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:12.015826    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 16:51:22.015399 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:22.015350    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 16:51:32.025338 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:32.025255    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:51:32.033239 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:32.033209    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:51:33.808994 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:33.808954    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"]
Sep 16 16:51:48.832543 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:48.832479    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main" containerID="cri-o://875b383d77888c90cce95bf5524f88930fbca9f3d227df70a2de8addbffb6355" gracePeriod=45
Sep 16 16:51:49.026424 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.026377    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 16:51:49.574247 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.574224    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:51:49.661323 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.661291    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-tmp-dir\") pod \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") "
Sep 16 16:51:49.661493 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.661349    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-home\") pod \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") "
Sep 16 16:51:49.661493 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.661390    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w4sh\" (UniqueName: \"kubernetes.io/projected/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-kube-api-access-2w4sh\") pod \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") "
Sep 16 16:51:49.661493 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.661428    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-tls-certs\") pod \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") "
Sep 16 16:51:49.661493 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.661452    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-cabundle-cert\") pod \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") "
Sep 16 16:51:49.661493 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.661480    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-dshm\") pod \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") "
Sep 16 16:51:49.661799 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.661552    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-kserve-provision-location\") pod \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") "
Sep 16 16:51:49.661799 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.661655    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-model-cache\") pod \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\" (UID: \"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6\") "
Sep 16 16:51:49.662002 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.661962    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" (UID: "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:51:49.662212 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.662192    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-home" (OuterVolumeSpecName: "home") pod "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" (UID: "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:51:49.662277 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.662208    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" (UID: "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:51:49.663727 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.663698    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" (UID: "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:51:49.663964 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.663937    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-kube-api-access-2w4sh" (OuterVolumeSpecName: "kube-api-access-2w4sh") pod "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" (UID: "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6"). InnerVolumeSpecName "kube-api-access-2w4sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:51:49.664096 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.664065    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-dshm" (OuterVolumeSpecName: "dshm") pod "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" (UID: "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:51:49.673197 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.673174    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" (UID: "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:51:49.688107 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.688084    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" (UID: "5e6f669b-eef8-4b22-a5ed-46c7cf4671a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:51:49.762673 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.762650    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:51:49.762673 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.762670    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:51:49.762793 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.762680    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w4sh\" (UniqueName: \"kubernetes.io/projected/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-kube-api-access-2w4sh\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:51:49.762793 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.762690    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:51:49.762793 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.762697    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:51:49.762793 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.762705    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:51:49.762793 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.762713    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:51:49.762793 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:49.762721    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:51:50.139936 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:50.139898    2574 generic.go:356] "Generic (PLEG): container finished" podID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerID="875b383d77888c90cce95bf5524f88930fbca9f3d227df70a2de8addbffb6355" exitCode=0
Sep 16 16:51:50.140305 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:50.139960    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" event={"ID":"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6","Type":"ContainerDied","Data":"875b383d77888c90cce95bf5524f88930fbca9f3d227df70a2de8addbffb6355"}
Sep 16 16:51:50.140305 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:50.140003    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg" event={"ID":"5e6f669b-eef8-4b22-a5ed-46c7cf4671a6","Type":"ContainerDied","Data":"8f38b65fc7800808c55133a83ef89996cd4028dff21d12da02faf41ea3e103fc"}
Sep 16 16:51:50.140305 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:50.140025    2574 scope.go:117] "RemoveContainer" containerID="875b383d77888c90cce95bf5524f88930fbca9f3d227df70a2de8addbffb6355"
Sep 16 16:51:50.140305 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:50.139965    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"
Sep 16 16:51:50.148443 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:50.148426    2574 scope.go:117] "RemoveContainer" containerID="1398dae09afb8fc8ad40975f177e4940b64ee5760e8d9798ccb4bcc393080984"
Sep 16 16:51:50.158346 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:50.158324    2574 scope.go:117] "RemoveContainer" containerID="875b383d77888c90cce95bf5524f88930fbca9f3d227df70a2de8addbffb6355"
Sep 16 16:51:50.158941 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:51:50.158916    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875b383d77888c90cce95bf5524f88930fbca9f3d227df70a2de8addbffb6355\": container with ID starting with 875b383d77888c90cce95bf5524f88930fbca9f3d227df70a2de8addbffb6355 not found: ID does not exist" containerID="875b383d77888c90cce95bf5524f88930fbca9f3d227df70a2de8addbffb6355"
Sep 16 16:51:50.159020 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:50.158946    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875b383d77888c90cce95bf5524f88930fbca9f3d227df70a2de8addbffb6355"} err="failed to get container status \"875b383d77888c90cce95bf5524f88930fbca9f3d227df70a2de8addbffb6355\": rpc error: code = NotFound desc = could not find container \"875b383d77888c90cce95bf5524f88930fbca9f3d227df70a2de8addbffb6355\": container with ID starting with 875b383d77888c90cce95bf5524f88930fbca9f3d227df70a2de8addbffb6355 not found: ID does not exist"
Sep 16 16:51:50.159020 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:50.158968    2574 scope.go:117] "RemoveContainer" containerID="1398dae09afb8fc8ad40975f177e4940b64ee5760e8d9798ccb4bcc393080984"
Sep 16 16:51:50.159283 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:51:50.159255    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1398dae09afb8fc8ad40975f177e4940b64ee5760e8d9798ccb4bcc393080984\": container with ID starting with 1398dae09afb8fc8ad40975f177e4940b64ee5760e8d9798ccb4bcc393080984 not found: ID does not exist" containerID="1398dae09afb8fc8ad40975f177e4940b64ee5760e8d9798ccb4bcc393080984"
Sep 16 16:51:50.159424 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:50.159392    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1398dae09afb8fc8ad40975f177e4940b64ee5760e8d9798ccb4bcc393080984"} err="failed to get container status \"1398dae09afb8fc8ad40975f177e4940b64ee5760e8d9798ccb4bcc393080984\": rpc error: code = NotFound desc = could not find container \"1398dae09afb8fc8ad40975f177e4940b64ee5760e8d9798ccb4bcc393080984\": container with ID starting with 1398dae09afb8fc8ad40975f177e4940b64ee5760e8d9798ccb4bcc393080984 not found: ID does not exist"
Sep 16 16:51:50.160828 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:50.160808    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"]
Sep 16 16:51:50.162559 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:50.162538    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-dc4484968-sf7rg"]
Sep 16 16:51:51.995589 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:51:51.995552    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" path="/var/lib/kubelet/pods/5e6f669b-eef8-4b22-a5ed-46c7cf4671a6/volumes"
Sep 16 16:52:38.088975 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.088944    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"]
Sep 16 16:52:38.089457 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.089328    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="storage-initializer"
Sep 16 16:52:38.089457 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.089340    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="storage-initializer"
Sep 16 16:52:38.089457 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.089349    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main"
Sep 16 16:52:38.089457 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.089354    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main"
Sep 16 16:52:38.089457 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.089409    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e6f669b-eef8-4b22-a5ed-46c7cf4671a6" containerName="main"
Sep 16 16:52:38.092816 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.092795    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.094908 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.094887    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:52:38.095075 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.095054    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 16 16:52:38.095213 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.095195    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-4mr58\""
Sep 16 16:52:38.095292 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.095273    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 16 16:52:38.095292 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.095285    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 16 16:52:38.095470 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.095445    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 16 16:52:38.104176 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.104155    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"]
Sep 16 16:52:38.161392 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.161366    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-748fn\" (UniqueName: \"kubernetes.io/projected/331818d8-430e-4356-b870-39bf7c8ff887-kube-api-access-748fn\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.161486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.161413    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-model-cache\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.161486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.161460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-tmp-dir\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.161612 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.161496    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-home\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.161612 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.161512    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-dshm\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.161612 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.161587    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.161728 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.161617    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/331818d8-430e-4356-b870-39bf7c8ff887-cabundle-cert\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.161728 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.161638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/331818d8-430e-4356-b870-39bf7c8ff887-tls-certs\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.262836 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.262803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-model-cache\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.262994 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.262850    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-tmp-dir\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.262994 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.262885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-home\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.262994 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.262905    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-dshm\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.262994 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.262955    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.263202 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.263071    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/331818d8-430e-4356-b870-39bf7c8ff887-cabundle-cert\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.263202 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.263105    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/331818d8-430e-4356-b870-39bf7c8ff887-tls-certs\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.263202 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.263164    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-748fn\" (UniqueName: \"kubernetes.io/projected/331818d8-430e-4356-b870-39bf7c8ff887-kube-api-access-748fn\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.263352 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.263241    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-model-cache\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.263352 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.263268    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-tmp-dir\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.263447 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.263376    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.263447 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.263396    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-home\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.263873 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.263849    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/331818d8-430e-4356-b870-39bf7c8ff887-cabundle-cert\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.265352 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.265318    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-dshm\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.265610 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.265592    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/331818d8-430e-4356-b870-39bf7c8ff887-tls-certs\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.270858 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.270835    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-748fn\" (UniqueName: \"kubernetes.io/projected/331818d8-430e-4356-b870-39bf7c8ff887-kube-api-access-748fn\") pod \"lora-single-adapter-test-kserve-dbcf6798b-5wfs2\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.403532 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.403487    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:38.689435 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:38.689405    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"]
Sep 16 16:52:38.690652 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:52:38.690629    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331818d8_430e_4356_b870_39bf7c8ff887.slice/crio-1bc4d0b928df52fe67ecf24d93232107feb86f438ea7ffc26e1294b83a1e6d49 WatchSource:0}: Error finding container 1bc4d0b928df52fe67ecf24d93232107feb86f438ea7ffc26e1294b83a1e6d49: Status 404 returned error can't find the container with id 1bc4d0b928df52fe67ecf24d93232107feb86f438ea7ffc26e1294b83a1e6d49
Sep 16 16:52:39.370821 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:39.370786    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" event={"ID":"331818d8-430e-4356-b870-39bf7c8ff887","Type":"ContainerStarted","Data":"d1d1600c677d2e466f8f29a6643c55fc7bb723971081e10d5096f13cd07fb0b2"}
Sep 16 16:52:39.371188 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:39.370842    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" event={"ID":"331818d8-430e-4356-b870-39bf7c8ff887","Type":"ContainerStarted","Data":"1bc4d0b928df52fe67ecf24d93232107feb86f438ea7ffc26e1294b83a1e6d49"}
Sep 16 16:52:42.388062 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:42.388030    2574 generic.go:356] "Generic (PLEG): container finished" podID="331818d8-430e-4356-b870-39bf7c8ff887" containerID="d1d1600c677d2e466f8f29a6643c55fc7bb723971081e10d5096f13cd07fb0b2" exitCode=0
Sep 16 16:52:42.388449 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:42.388112    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" event={"ID":"331818d8-430e-4356-b870-39bf7c8ff887","Type":"ContainerDied","Data":"d1d1600c677d2e466f8f29a6643c55fc7bb723971081e10d5096f13cd07fb0b2"}
Sep 16 16:52:43.395264 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:43.395220    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" event={"ID":"331818d8-430e-4356-b870-39bf7c8ff887","Type":"ContainerStarted","Data":"10fb7c6e9226f48fc60aaae5e134edfd8782e7909ac07e5be082067c32353734"}
Sep 16 16:52:43.404490 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:43.404465    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:43.419444 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:43.419396    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" podStartSLOduration=2.706810149 podStartE2EDuration="5.419379345s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.712569196s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:52:43.417587273 +0000 UTC m=+3759.937210049" watchObservedRunningTime="2026-09-16 16:52:43.419379345 +0000 UTC m=+3759.939002123"
Sep 16 16:52:48.403807 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:48.403769    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:52:48.405065 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:48.405033    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" podUID="331818d8-430e-4356-b870-39bf7c8ff887" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 16:52:51.345840 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:51.345804    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"]
Sep 16 16:52:57.781453 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:57.781370    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt" podUID="712e8481-0875-424b-94d9-e5f06444993f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:52:58.404893 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:52:58.404847    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" podUID="331818d8-430e-4356-b870-39bf7c8ff887" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 16:53:06.365611 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:06.365555    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt" podUID="712e8481-0875-424b-94d9-e5f06444993f" containerName="main" containerID="cri-o://3d71a543aeda3a51c220a19101b6756568521dc13e74016daa35913d135cd6e8" gracePeriod=45
Sep 16 16:53:07.781022 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:07.780977    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt" podUID="712e8481-0875-424b-94d9-e5f06444993f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:53:08.403931 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:08.403887    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" podUID="331818d8-430e-4356-b870-39bf7c8ff887" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 16:53:17.781060 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:17.781020    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt" podUID="712e8481-0875-424b-94d9-e5f06444993f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:53:17.781508 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:17.781130    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"
Sep 16 16:53:18.404583 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:18.404511    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" podUID="331818d8-430e-4356-b870-39bf7c8ff887" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 16:53:27.780568 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:27.780490    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt" podUID="712e8481-0875-424b-94d9-e5f06444993f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:53:28.403917 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:28.403870    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" podUID="331818d8-430e-4356-b870-39bf7c8ff887" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 16:53:36.615980 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.615949    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"
Sep 16 16:53:36.667503 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.667411    2574 generic.go:356] "Generic (PLEG): container finished" podID="712e8481-0875-424b-94d9-e5f06444993f" containerID="3d71a543aeda3a51c220a19101b6756568521dc13e74016daa35913d135cd6e8" exitCode=0
Sep 16 16:53:36.667503 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.667472    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"
Sep 16 16:53:36.667721 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.667478    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt" event={"ID":"712e8481-0875-424b-94d9-e5f06444993f","Type":"ContainerDied","Data":"3d71a543aeda3a51c220a19101b6756568521dc13e74016daa35913d135cd6e8"}
Sep 16 16:53:36.667721 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.667582    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt" event={"ID":"712e8481-0875-424b-94d9-e5f06444993f","Type":"ContainerDied","Data":"9c35a4c67d9af1ba63919aba2293ae69e60b7715d4d183be892d9e48b496898b"}
Sep 16 16:53:36.667721 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.667615    2574 scope.go:117] "RemoveContainer" containerID="3d71a543aeda3a51c220a19101b6756568521dc13e74016daa35913d135cd6e8"
Sep 16 16:53:36.677437 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.677416    2574 scope.go:117] "RemoveContainer" containerID="3d71a543aeda3a51c220a19101b6756568521dc13e74016daa35913d135cd6e8"
Sep 16 16:53:36.677739 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:53:36.677716    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d71a543aeda3a51c220a19101b6756568521dc13e74016daa35913d135cd6e8\": container with ID starting with 3d71a543aeda3a51c220a19101b6756568521dc13e74016daa35913d135cd6e8 not found: ID does not exist" containerID="3d71a543aeda3a51c220a19101b6756568521dc13e74016daa35913d135cd6e8"
Sep 16 16:53:36.677802 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.677749    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d71a543aeda3a51c220a19101b6756568521dc13e74016daa35913d135cd6e8"} err="failed to get container status \"3d71a543aeda3a51c220a19101b6756568521dc13e74016daa35913d135cd6e8\": rpc error: code = NotFound desc = could not find container \"3d71a543aeda3a51c220a19101b6756568521dc13e74016daa35913d135cd6e8\": container with ID starting with 3d71a543aeda3a51c220a19101b6756568521dc13e74016daa35913d135cd6e8 not found: ID does not exist"
Sep 16 16:53:36.707110 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.707075    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f6n8\" (UniqueName: \"kubernetes.io/projected/712e8481-0875-424b-94d9-e5f06444993f-kube-api-access-5f6n8\") pod \"712e8481-0875-424b-94d9-e5f06444993f\" (UID: \"712e8481-0875-424b-94d9-e5f06444993f\") "
Sep 16 16:53:36.707276 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.707128    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712e8481-0875-424b-94d9-e5f06444993f-tls-certs\") pod \"712e8481-0875-424b-94d9-e5f06444993f\" (UID: \"712e8481-0875-424b-94d9-e5f06444993f\") "
Sep 16 16:53:36.709301 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.709274    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712e8481-0875-424b-94d9-e5f06444993f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "712e8481-0875-424b-94d9-e5f06444993f" (UID: "712e8481-0875-424b-94d9-e5f06444993f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:53:36.709426 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.709399    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712e8481-0875-424b-94d9-e5f06444993f-kube-api-access-5f6n8" (OuterVolumeSpecName: "kube-api-access-5f6n8") pod "712e8481-0875-424b-94d9-e5f06444993f" (UID: "712e8481-0875-424b-94d9-e5f06444993f"). InnerVolumeSpecName "kube-api-access-5f6n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:53:36.808715 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.808667    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f6n8\" (UniqueName: \"kubernetes.io/projected/712e8481-0875-424b-94d9-e5f06444993f-kube-api-access-5f6n8\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:53:36.808715 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.808701    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712e8481-0875-424b-94d9-e5f06444993f-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:53:36.992146 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.992100    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"]
Sep 16 16:53:36.996337 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:36.996309    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58lb7bt"]
Sep 16 16:53:37.995223 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:37.995187    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="712e8481-0875-424b-94d9-e5f06444993f" path="/var/lib/kubelet/pods/712e8481-0875-424b-94d9-e5f06444993f/volumes"
Sep 16 16:53:38.404104 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:38.404049    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" podUID="331818d8-430e-4356-b870-39bf7c8ff887" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 16:53:48.404553 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:48.404489    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" podUID="331818d8-430e-4356-b870-39bf7c8ff887" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 16 16:53:58.413801 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.413762    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:53:58.422187 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.422159    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:53:58.553054 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.553016    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"]
Sep 16 16:53:58.553437 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.553421    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="712e8481-0875-424b-94d9-e5f06444993f" containerName="main"
Sep 16 16:53:58.553437 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.553436    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="712e8481-0875-424b-94d9-e5f06444993f" containerName="main"
Sep 16 16:53:58.553582 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.553510    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="712e8481-0875-424b-94d9-e5f06444993f" containerName="main"
Sep 16 16:53:58.556448 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.556430    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"
Sep 16 16:53:58.558795 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.558763    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 16:53:58.558923 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.558856    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-fvtfr\""
Sep 16 16:53:58.558923 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.558905    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 16:53:58.559128 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.559103    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-4hrzn\""
Sep 16 16:53:58.559240 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.559169    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 16:53:58.564041 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.564017    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"]
Sep 16 16:53:58.608681 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.608644    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqk8f\" (UniqueName: \"kubernetes.io/projected/7f1e9fc4-7cbc-4148-8407-19f22915f4dd-kube-api-access-vqk8f\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr\" (UID: \"7f1e9fc4-7cbc-4148-8407-19f22915f4dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"
Sep 16 16:53:58.608844 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.608719    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1e9fc4-7cbc-4148-8407-19f22915f4dd-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr\" (UID: \"7f1e9fc4-7cbc-4148-8407-19f22915f4dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"
Sep 16 16:53:58.709757 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.709653    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1e9fc4-7cbc-4148-8407-19f22915f4dd-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr\" (UID: \"7f1e9fc4-7cbc-4148-8407-19f22915f4dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"
Sep 16 16:53:58.709938 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.709779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk8f\" (UniqueName: \"kubernetes.io/projected/7f1e9fc4-7cbc-4148-8407-19f22915f4dd-kube-api-access-vqk8f\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr\" (UID: \"7f1e9fc4-7cbc-4148-8407-19f22915f4dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"
Sep 16 16:53:58.712198 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.712175    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1e9fc4-7cbc-4148-8407-19f22915f4dd-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr\" (UID: \"7f1e9fc4-7cbc-4148-8407-19f22915f4dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"
Sep 16 16:53:58.718894 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.718871    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqk8f\" (UniqueName: \"kubernetes.io/projected/7f1e9fc4-7cbc-4148-8407-19f22915f4dd-kube-api-access-vqk8f\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr\" (UID: \"7f1e9fc4-7cbc-4148-8407-19f22915f4dd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"
Sep 16 16:53:58.868441 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:58.868404    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"
Sep 16 16:53:59.138713 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:59.138685    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"]
Sep 16 16:53:59.140894 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:53:59.140864    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1e9fc4_7cbc_4148_8407_19f22915f4dd.slice/crio-c6ff53a90d9c8df10926076c792b1fabdbfae874b9c38bb930239bc21df3325e WatchSource:0}: Error finding container c6ff53a90d9c8df10926076c792b1fabdbfae874b9c38bb930239bc21df3325e: Status 404 returned error can't find the container with id c6ff53a90d9c8df10926076c792b1fabdbfae874b9c38bb930239bc21df3325e
Sep 16 16:53:59.796552 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:59.796495    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr" event={"ID":"7f1e9fc4-7cbc-4148-8407-19f22915f4dd","Type":"ContainerStarted","Data":"5b316b4294cb130a2f3f63d5e53ac43df4237dd406cb83758487592f247b1fac"}
Sep 16 16:53:59.796922 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:59.796668    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"
Sep 16 16:53:59.796922 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:59.796695    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr" event={"ID":"7f1e9fc4-7cbc-4148-8407-19f22915f4dd","Type":"ContainerStarted","Data":"c6ff53a90d9c8df10926076c792b1fabdbfae874b9c38bb930239bc21df3325e"}
Sep 16 16:53:59.812687 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:53:59.812634    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr" podStartSLOduration=1.8126193659999998 podStartE2EDuration="1.812619366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:53:59.8114395 +0000 UTC m=+3836.331062276" watchObservedRunningTime="2026-09-16 16:53:59.812619366 +0000 UTC m=+3836.332242141"
Sep 16 16:54:30.804882 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:30.804800    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"
Sep 16 16:54:39.292722 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:39.292672    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-dbcf6798b-5wfs2_331818d8-430e-4356-b870-39bf7c8ff887/storage-initializer/0.log"
Sep 16 16:54:39.301917 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:39.301882    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-dbcf6798b-5wfs2_331818d8-430e-4356-b870-39bf7c8ff887/main/0.log"
Sep 16 16:54:40.614529 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:40.614488    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"]
Sep 16 16:54:55.637046 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:55.637000    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" podUID="331818d8-430e-4356-b870-39bf7c8ff887" containerName="main" containerID="cri-o://10fb7c6e9226f48fc60aaae5e134edfd8782e7909ac07e5be082067c32353734" gracePeriod=45
Sep 16 16:54:56.279712 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.279687    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:54:56.414627 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.414598    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/331818d8-430e-4356-b870-39bf7c8ff887-tls-certs\") pod \"331818d8-430e-4356-b870-39bf7c8ff887\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") "
Sep 16 16:54:56.414627 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.414631    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-dshm\") pod \"331818d8-430e-4356-b870-39bf7c8ff887\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") "
Sep 16 16:54:56.414880 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.414651    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-tmp-dir\") pod \"331818d8-430e-4356-b870-39bf7c8ff887\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") "
Sep 16 16:54:56.414880 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.414674    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-748fn\" (UniqueName: \"kubernetes.io/projected/331818d8-430e-4356-b870-39bf7c8ff887-kube-api-access-748fn\") pod \"331818d8-430e-4356-b870-39bf7c8ff887\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") "
Sep 16 16:54:56.414880 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.414691    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/331818d8-430e-4356-b870-39bf7c8ff887-cabundle-cert\") pod \"331818d8-430e-4356-b870-39bf7c8ff887\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") "
Sep 16 16:54:56.414880 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.414731    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-home\") pod \"331818d8-430e-4356-b870-39bf7c8ff887\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") "
Sep 16 16:54:56.414880 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.414768    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-kserve-provision-location\") pod \"331818d8-430e-4356-b870-39bf7c8ff887\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") "
Sep 16 16:54:56.414880 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.414858    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-model-cache\") pod \"331818d8-430e-4356-b870-39bf7c8ff887\" (UID: \"331818d8-430e-4356-b870-39bf7c8ff887\") "
Sep 16 16:54:56.415252 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.415223    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-home" (OuterVolumeSpecName: "home") pod "331818d8-430e-4356-b870-39bf7c8ff887" (UID: "331818d8-430e-4356-b870-39bf7c8ff887"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:54:56.415656 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.415388    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-model-cache" (OuterVolumeSpecName: "model-cache") pod "331818d8-430e-4356-b870-39bf7c8ff887" (UID: "331818d8-430e-4356-b870-39bf7c8ff887"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:54:56.415656 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.415470    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331818d8-430e-4356-b870-39bf7c8ff887-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "331818d8-430e-4356-b870-39bf7c8ff887" (UID: "331818d8-430e-4356-b870-39bf7c8ff887"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:54:56.417169 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.417140    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331818d8-430e-4356-b870-39bf7c8ff887-kube-api-access-748fn" (OuterVolumeSpecName: "kube-api-access-748fn") pod "331818d8-430e-4356-b870-39bf7c8ff887" (UID: "331818d8-430e-4356-b870-39bf7c8ff887"). InnerVolumeSpecName "kube-api-access-748fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:54:56.417420 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.417398    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-dshm" (OuterVolumeSpecName: "dshm") pod "331818d8-430e-4356-b870-39bf7c8ff887" (UID: "331818d8-430e-4356-b870-39bf7c8ff887"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:54:56.417499 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.417413    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/331818d8-430e-4356-b870-39bf7c8ff887-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "331818d8-430e-4356-b870-39bf7c8ff887" (UID: "331818d8-430e-4356-b870-39bf7c8ff887"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:54:56.426285 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.426262    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "331818d8-430e-4356-b870-39bf7c8ff887" (UID: "331818d8-430e-4356-b870-39bf7c8ff887"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:54:56.439262 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.439240    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "331818d8-430e-4356-b870-39bf7c8ff887" (UID: "331818d8-430e-4356-b870-39bf7c8ff887"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:54:56.516344 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.516321    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/331818d8-430e-4356-b870-39bf7c8ff887-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:54:56.516344 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.516342    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:54:56.516486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.516351    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:54:56.516486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.516359    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-748fn\" (UniqueName: \"kubernetes.io/projected/331818d8-430e-4356-b870-39bf7c8ff887-kube-api-access-748fn\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:54:56.516486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.516370    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/331818d8-430e-4356-b870-39bf7c8ff887-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:54:56.516486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.516380    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:54:56.516486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.516387    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:54:56.516486 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:56.516396    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/331818d8-430e-4356-b870-39bf7c8ff887-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:54:57.081972 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:57.081942    2574 generic.go:356] "Generic (PLEG): container finished" podID="331818d8-430e-4356-b870-39bf7c8ff887" containerID="10fb7c6e9226f48fc60aaae5e134edfd8782e7909ac07e5be082067c32353734" exitCode=0
Sep 16 16:54:57.082386 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:57.081986    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" event={"ID":"331818d8-430e-4356-b870-39bf7c8ff887","Type":"ContainerDied","Data":"10fb7c6e9226f48fc60aaae5e134edfd8782e7909ac07e5be082067c32353734"}
Sep 16 16:54:57.082386 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:57.082024    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2" event={"ID":"331818d8-430e-4356-b870-39bf7c8ff887","Type":"ContainerDied","Data":"1bc4d0b928df52fe67ecf24d93232107feb86f438ea7ffc26e1294b83a1e6d49"}
Sep 16 16:54:57.082386 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:57.082044    2574 scope.go:117] "RemoveContainer" containerID="10fb7c6e9226f48fc60aaae5e134edfd8782e7909ac07e5be082067c32353734"
Sep 16 16:54:57.082386 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:57.081990    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"
Sep 16 16:54:57.091950 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:57.091787    2574 scope.go:117] "RemoveContainer" containerID="d1d1600c677d2e466f8f29a6643c55fc7bb723971081e10d5096f13cd07fb0b2"
Sep 16 16:54:57.099715 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:57.099698    2574 scope.go:117] "RemoveContainer" containerID="10fb7c6e9226f48fc60aaae5e134edfd8782e7909ac07e5be082067c32353734"
Sep 16 16:54:57.099966 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:54:57.099948    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10fb7c6e9226f48fc60aaae5e134edfd8782e7909ac07e5be082067c32353734\": container with ID starting with 10fb7c6e9226f48fc60aaae5e134edfd8782e7909ac07e5be082067c32353734 not found: ID does not exist" containerID="10fb7c6e9226f48fc60aaae5e134edfd8782e7909ac07e5be082067c32353734"
Sep 16 16:54:57.100027 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:57.099972    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10fb7c6e9226f48fc60aaae5e134edfd8782e7909ac07e5be082067c32353734"} err="failed to get container status \"10fb7c6e9226f48fc60aaae5e134edfd8782e7909ac07e5be082067c32353734\": rpc error: code = NotFound desc = could not find container \"10fb7c6e9226f48fc60aaae5e134edfd8782e7909ac07e5be082067c32353734\": container with ID starting with 10fb7c6e9226f48fc60aaae5e134edfd8782e7909ac07e5be082067c32353734 not found: ID does not exist"
Sep 16 16:54:57.100027 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:57.099987    2574 scope.go:117] "RemoveContainer" containerID="d1d1600c677d2e466f8f29a6643c55fc7bb723971081e10d5096f13cd07fb0b2"
Sep 16 16:54:57.100231 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:54:57.100215    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1d1600c677d2e466f8f29a6643c55fc7bb723971081e10d5096f13cd07fb0b2\": container with ID starting with d1d1600c677d2e466f8f29a6643c55fc7bb723971081e10d5096f13cd07fb0b2 not found: ID does not exist" containerID="d1d1600c677d2e466f8f29a6643c55fc7bb723971081e10d5096f13cd07fb0b2"
Sep 16 16:54:57.100282 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:57.100236    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1d1600c677d2e466f8f29a6643c55fc7bb723971081e10d5096f13cd07fb0b2"} err="failed to get container status \"d1d1600c677d2e466f8f29a6643c55fc7bb723971081e10d5096f13cd07fb0b2\": rpc error: code = NotFound desc = could not find container \"d1d1600c677d2e466f8f29a6643c55fc7bb723971081e10d5096f13cd07fb0b2\": container with ID starting with d1d1600c677d2e466f8f29a6643c55fc7bb723971081e10d5096f13cd07fb0b2 not found: ID does not exist"
Sep 16 16:54:57.110511 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:57.110488    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"]
Sep 16 16:54:57.121345 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:57.121322    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-dbcf6798b-5wfs2"]
Sep 16 16:54:57.993968 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:54:57.993935    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="331818d8-430e-4356-b870-39bf7c8ff887" path="/var/lib/kubelet/pods/331818d8-430e-4356-b870-39bf7c8ff887/volumes"
Sep 16 16:55:04.287029 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.286998    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"]
Sep 16 16:55:04.287505 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.287374    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="331818d8-430e-4356-b870-39bf7c8ff887" containerName="main"
Sep 16 16:55:04.287505 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.287386    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="331818d8-430e-4356-b870-39bf7c8ff887" containerName="main"
Sep 16 16:55:04.287505 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.287416    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="331818d8-430e-4356-b870-39bf7c8ff887" containerName="storage-initializer"
Sep 16 16:55:04.287505 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.287422    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="331818d8-430e-4356-b870-39bf7c8ff887" containerName="storage-initializer"
Sep 16 16:55:04.287505 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.287489    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="331818d8-430e-4356-b870-39bf7c8ff887" containerName="main"
Sep 16 16:55:04.291986 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.291961    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.294491 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.294470    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-lzgrf\""
Sep 16 16:55:04.294746 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.294485    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 16 16:55:04.294976 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.294489    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 16 16:55:04.294976 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.294553    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 16 16:55:04.294976 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.294564    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 16 16:55:04.294976 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.294572    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:55:04.305643 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.305616    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"]
Sep 16 16:55:04.382077 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.382045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.382077 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.382076    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46fcb879-e90d-4b88-8ca8-51d040dfaf17-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.382271 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.382095    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-model-cache\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.382271 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.382216    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-dshm\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.382271 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.382251    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llkp6\" (UniqueName: \"kubernetes.io/projected/46fcb879-e90d-4b88-8ca8-51d040dfaf17-kube-api-access-llkp6\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.382271 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.382266    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46fcb879-e90d-4b88-8ca8-51d040dfaf17-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.382406 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.382284    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.382406 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.382318    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-home\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.483201 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.483175    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-dshm\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.483340 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.483217    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llkp6\" (UniqueName: \"kubernetes.io/projected/46fcb879-e90d-4b88-8ca8-51d040dfaf17-kube-api-access-llkp6\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.483340 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.483236    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46fcb879-e90d-4b88-8ca8-51d040dfaf17-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.483340 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.483260    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.483340 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.483285    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-home\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.483561 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.483461    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.483561 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.483506    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46fcb879-e90d-4b88-8ca8-51d040dfaf17-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.483688 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.483664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-model-cache\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.483745 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.483707    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.483745 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.483727    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-home\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.483928 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.483902    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.484048 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.483983    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-model-cache\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.484189 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.484170    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46fcb879-e90d-4b88-8ca8-51d040dfaf17-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.485269 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.485248    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-dshm\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.485725 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.485707    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46fcb879-e90d-4b88-8ca8-51d040dfaf17-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.493290 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.493266    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llkp6\" (UniqueName: \"kubernetes.io/projected/46fcb879-e90d-4b88-8ca8-51d040dfaf17-kube-api-access-llkp6\") pod \"lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.604631 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.604604    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:04.879712 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:04.879688    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"]
Sep 16 16:55:04.882677 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:55:04.882648    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fcb879_e90d_4b88_8ca8_51d040dfaf17.slice/crio-07d43fa5c18c99fd6537e99e7f8e3489cd3a278d80104d99fd5ede3d4a462fa8 WatchSource:0}: Error finding container 07d43fa5c18c99fd6537e99e7f8e3489cd3a278d80104d99fd5ede3d4a462fa8: Status 404 returned error can't find the container with id 07d43fa5c18c99fd6537e99e7f8e3489cd3a278d80104d99fd5ede3d4a462fa8
Sep 16 16:55:05.119605 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:05.119564    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" event={"ID":"46fcb879-e90d-4b88-8ca8-51d040dfaf17","Type":"ContainerStarted","Data":"9d370ca142a5b336527ee59867e1bf2520c7d261e5eaa11627185241331bfec2"}
Sep 16 16:55:05.119784 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:05.119621    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" event={"ID":"46fcb879-e90d-4b88-8ca8-51d040dfaf17","Type":"ContainerStarted","Data":"07d43fa5c18c99fd6537e99e7f8e3489cd3a278d80104d99fd5ede3d4a462fa8"}
Sep 16 16:55:08.139929 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:08.139895    2574 generic.go:356] "Generic (PLEG): container finished" podID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerID="9d370ca142a5b336527ee59867e1bf2520c7d261e5eaa11627185241331bfec2" exitCode=0
Sep 16 16:55:08.140390 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:08.139966    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" event={"ID":"46fcb879-e90d-4b88-8ca8-51d040dfaf17","Type":"ContainerDied","Data":"9d370ca142a5b336527ee59867e1bf2520c7d261e5eaa11627185241331bfec2"}
Sep 16 16:55:09.146355 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:09.146318    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" event={"ID":"46fcb879-e90d-4b88-8ca8-51d040dfaf17","Type":"ContainerStarted","Data":"2dd461997d2bb59b6fc9a814fc1a5181d2de0f7b72cbd7cfc660cb3f519ae278"}
Sep 16 16:55:09.169425 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:09.169363    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" podStartSLOduration=2.20626756 podStartE2EDuration="5.169342549s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.963074989s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:55:09.167721543 +0000 UTC m=+3905.687344321" watchObservedRunningTime="2026-09-16 16:55:09.169342549 +0000 UTC m=+3905.688965326"
Sep 16 16:55:09.605200 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:09.605163    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:14.605696 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:14.605657    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:55:14.606895 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:14.606865    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 16:55:24.605847 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:24.605802    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 16:55:34.605505 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:34.605465    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 16:55:44.605782 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:44.605734    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 16:55:54.605307 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:55:54.605268    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 16:56:03.603974 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:03.603934    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"]
Sep 16 16:56:04.605300 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:04.605246    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 16:56:10.804292 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:10.804249    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr" podUID="7f1e9fc4-7cbc-4148-8407-19f22915f4dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:56:14.605079 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:14.605025    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 16:56:18.626278 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:18.626237    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr" podUID="7f1e9fc4-7cbc-4148-8407-19f22915f4dd" containerName="main" containerID="cri-o://5b316b4294cb130a2f3f63d5e53ac43df4237dd406cb83758487592f247b1fac" gracePeriod=45
Sep 16 16:56:20.803967 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:20.803929    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr" podUID="7f1e9fc4-7cbc-4148-8407-19f22915f4dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:56:24.614784 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:24.614752    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:56:24.622087 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:24.622058    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:56:30.804497 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:30.804454    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr" podUID="7f1e9fc4-7cbc-4148-8407-19f22915f4dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:56:30.805000 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:30.804583    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"
Sep 16 16:56:40.803995 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:40.803952    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr" podUID="7f1e9fc4-7cbc-4148-8407-19f22915f4dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 16:56:48.906572 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:48.906542    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"
Sep 16 16:56:48.966876 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:48.966839    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqk8f\" (UniqueName: \"kubernetes.io/projected/7f1e9fc4-7cbc-4148-8407-19f22915f4dd-kube-api-access-vqk8f\") pod \"7f1e9fc4-7cbc-4148-8407-19f22915f4dd\" (UID: \"7f1e9fc4-7cbc-4148-8407-19f22915f4dd\") "
Sep 16 16:56:48.967021 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:48.966935    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1e9fc4-7cbc-4148-8407-19f22915f4dd-tls-certs\") pod \"7f1e9fc4-7cbc-4148-8407-19f22915f4dd\" (UID: \"7f1e9fc4-7cbc-4148-8407-19f22915f4dd\") "
Sep 16 16:56:48.969235 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:48.969197    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f1e9fc4-7cbc-4148-8407-19f22915f4dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f1e9fc4-7cbc-4148-8407-19f22915f4dd" (UID: "7f1e9fc4-7cbc-4148-8407-19f22915f4dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:56:48.969377 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:48.969253    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1e9fc4-7cbc-4148-8407-19f22915f4dd-kube-api-access-vqk8f" (OuterVolumeSpecName: "kube-api-access-vqk8f") pod "7f1e9fc4-7cbc-4148-8407-19f22915f4dd" (UID: "7f1e9fc4-7cbc-4148-8407-19f22915f4dd"). InnerVolumeSpecName "kube-api-access-vqk8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:56:49.068682 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:49.068592    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqk8f\" (UniqueName: \"kubernetes.io/projected/7f1e9fc4-7cbc-4148-8407-19f22915f4dd-kube-api-access-vqk8f\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:56:49.068682 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:49.068628    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f1e9fc4-7cbc-4148-8407-19f22915f4dd-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:56:49.665259 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:49.665223    2574 generic.go:356] "Generic (PLEG): container finished" podID="7f1e9fc4-7cbc-4148-8407-19f22915f4dd" containerID="5b316b4294cb130a2f3f63d5e53ac43df4237dd406cb83758487592f247b1fac" exitCode=0
Sep 16 16:56:49.665447 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:49.665269    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"
Sep 16 16:56:49.665447 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:49.665305    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr" event={"ID":"7f1e9fc4-7cbc-4148-8407-19f22915f4dd","Type":"ContainerDied","Data":"5b316b4294cb130a2f3f63d5e53ac43df4237dd406cb83758487592f247b1fac"}
Sep 16 16:56:49.665447 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:49.665359    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr" event={"ID":"7f1e9fc4-7cbc-4148-8407-19f22915f4dd","Type":"ContainerDied","Data":"c6ff53a90d9c8df10926076c792b1fabdbfae874b9c38bb930239bc21df3325e"}
Sep 16 16:56:49.665447 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:49.665387    2574 scope.go:117] "RemoveContainer" containerID="5b316b4294cb130a2f3f63d5e53ac43df4237dd406cb83758487592f247b1fac"
Sep 16 16:56:49.674212 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:49.674193    2574 scope.go:117] "RemoveContainer" containerID="5b316b4294cb130a2f3f63d5e53ac43df4237dd406cb83758487592f247b1fac"
Sep 16 16:56:49.674493 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:56:49.674470    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b316b4294cb130a2f3f63d5e53ac43df4237dd406cb83758487592f247b1fac\": container with ID starting with 5b316b4294cb130a2f3f63d5e53ac43df4237dd406cb83758487592f247b1fac not found: ID does not exist" containerID="5b316b4294cb130a2f3f63d5e53ac43df4237dd406cb83758487592f247b1fac"
Sep 16 16:56:49.674574 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:49.674503    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b316b4294cb130a2f3f63d5e53ac43df4237dd406cb83758487592f247b1fac"} err="failed to get container status \"5b316b4294cb130a2f3f63d5e53ac43df4237dd406cb83758487592f247b1fac\": rpc error: code = NotFound desc = could not find container \"5b316b4294cb130a2f3f63d5e53ac43df4237dd406cb83758487592f247b1fac\": container with ID starting with 5b316b4294cb130a2f3f63d5e53ac43df4237dd406cb83758487592f247b1fac not found: ID does not exist"
Sep 16 16:56:49.685739 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:49.685717    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"]
Sep 16 16:56:49.687297 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:49.687278    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-nhsxr"]
Sep 16 16:56:49.995165 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:49.995078    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f1e9fc4-7cbc-4148-8407-19f22915f4dd" path="/var/lib/kubelet/pods/7f1e9fc4-7cbc-4148-8407-19f22915f4dd/volumes"
Sep 16 16:56:50.881328 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:50.881288    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj_46fcb879-e90d-4b88-8ca8-51d040dfaf17/storage-initializer/0.log"
Sep 16 16:56:50.889861 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:50.889818    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj_46fcb879-e90d-4b88-8ca8-51d040dfaf17/main/0.log"
Sep 16 16:56:51.884915 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:56:51.884881    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"]
Sep 16 16:57:04.548806 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.548769    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"]
Sep 16 16:57:04.549992 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.549963    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f1e9fc4-7cbc-4148-8407-19f22915f4dd" containerName="main"
Sep 16 16:57:04.549992 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.549993    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f1e9fc4-7cbc-4148-8407-19f22915f4dd" containerName="main"
Sep 16 16:57:04.550141 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.550113    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f1e9fc4-7cbc-4148-8407-19f22915f4dd" containerName="main"
Sep 16 16:57:04.553421 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.553402    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"
Sep 16 16:57:04.556546 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.556513    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 16:57:04.556658 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.556527    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 16:57:04.556658 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.556634    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-lgp9k\""
Sep 16 16:57:04.556875 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.556845    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 16:57:04.557104 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.557084    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-6dzrl\""
Sep 16 16:57:04.562808 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.562784    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"]
Sep 16 16:57:04.711796 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.711747    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa87d418-dc20-4e57-874c-84ee472be47d-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx\" (UID: \"fa87d418-dc20-4e57-874c-84ee472be47d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"
Sep 16 16:57:04.711971 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.711867    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrpcw\" (UniqueName: \"kubernetes.io/projected/fa87d418-dc20-4e57-874c-84ee472be47d-kube-api-access-zrpcw\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx\" (UID: \"fa87d418-dc20-4e57-874c-84ee472be47d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"
Sep 16 16:57:04.812443 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.812350    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrpcw\" (UniqueName: \"kubernetes.io/projected/fa87d418-dc20-4e57-874c-84ee472be47d-kube-api-access-zrpcw\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx\" (UID: \"fa87d418-dc20-4e57-874c-84ee472be47d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"
Sep 16 16:57:04.812632 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.812479    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa87d418-dc20-4e57-874c-84ee472be47d-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx\" (UID: \"fa87d418-dc20-4e57-874c-84ee472be47d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"
Sep 16 16:57:04.815205 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.815181    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa87d418-dc20-4e57-874c-84ee472be47d-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx\" (UID: \"fa87d418-dc20-4e57-874c-84ee472be47d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"
Sep 16 16:57:04.820053 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.820034    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrpcw\" (UniqueName: \"kubernetes.io/projected/fa87d418-dc20-4e57-874c-84ee472be47d-kube-api-access-zrpcw\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx\" (UID: \"fa87d418-dc20-4e57-874c-84ee472be47d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"
Sep 16 16:57:04.865197 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:04.865135    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"
Sep 16 16:57:05.137839 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:05.137810    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"]
Sep 16 16:57:05.139789 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:57:05.139766    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa87d418_dc20_4e57_874c_84ee472be47d.slice/crio-ee55a2f0c6cccada991094ecbd84b3052402e45a8b0112510d15ec5e2dd9f5b8 WatchSource:0}: Error finding container ee55a2f0c6cccada991094ecbd84b3052402e45a8b0112510d15ec5e2dd9f5b8: Status 404 returned error can't find the container with id ee55a2f0c6cccada991094ecbd84b3052402e45a8b0112510d15ec5e2dd9f5b8
Sep 16 16:57:05.747528 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:05.747489    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx" event={"ID":"fa87d418-dc20-4e57-874c-84ee472be47d","Type":"ContainerStarted","Data":"148220f06e3598a3309775358a719f4edb7691b5b3070b47ceccdce0a976f3a3"}
Sep 16 16:57:05.747934 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:05.747566    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx" event={"ID":"fa87d418-dc20-4e57-874c-84ee472be47d","Type":"ContainerStarted","Data":"ee55a2f0c6cccada991094ecbd84b3052402e45a8b0112510d15ec5e2dd9f5b8"}
Sep 16 16:57:05.747934 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:05.747629    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"
Sep 16 16:57:05.764664 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:05.764610    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx" podStartSLOduration=1.7645950639999999 podStartE2EDuration="1.764595064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:57:05.762927338 +0000 UTC m=+4022.282550118" watchObservedRunningTime="2026-09-16 16:57:05.764595064 +0000 UTC m=+4022.284217839"
Sep 16 16:57:06.906617 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:06.906566    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerName="main" containerID="cri-o://2dd461997d2bb59b6fc9a814fc1a5181d2de0f7b72cbd7cfc660cb3f519ae278" gracePeriod=45
Sep 16 16:57:07.676643 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.676613    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:57:07.760013 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.759926    2574 generic.go:356] "Generic (PLEG): container finished" podID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerID="2dd461997d2bb59b6fc9a814fc1a5181d2de0f7b72cbd7cfc660cb3f519ae278" exitCode=0
Sep 16 16:57:07.760013 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.759981    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"
Sep 16 16:57:07.760191 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.759983    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" event={"ID":"46fcb879-e90d-4b88-8ca8-51d040dfaf17","Type":"ContainerDied","Data":"2dd461997d2bb59b6fc9a814fc1a5181d2de0f7b72cbd7cfc660cb3f519ae278"}
Sep 16 16:57:07.760191 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.760151    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" event={"ID":"46fcb879-e90d-4b88-8ca8-51d040dfaf17","Type":"ContainerDied","Data":"07d43fa5c18c99fd6537e99e7f8e3489cd3a278d80104d99fd5ede3d4a462fa8"}
Sep 16 16:57:07.760191 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.760188    2574 scope.go:117] "RemoveContainer" containerID="2dd461997d2bb59b6fc9a814fc1a5181d2de0f7b72cbd7cfc660cb3f519ae278"
Sep 16 16:57:07.768836 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.768812    2574 scope.go:117] "RemoveContainer" containerID="9d370ca142a5b336527ee59867e1bf2520c7d261e5eaa11627185241331bfec2"
Sep 16 16:57:07.776851 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.776829    2574 scope.go:117] "RemoveContainer" containerID="2dd461997d2bb59b6fc9a814fc1a5181d2de0f7b72cbd7cfc660cb3f519ae278"
Sep 16 16:57:07.777182 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:57:07.777156    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd461997d2bb59b6fc9a814fc1a5181d2de0f7b72cbd7cfc660cb3f519ae278\": container with ID starting with 2dd461997d2bb59b6fc9a814fc1a5181d2de0f7b72cbd7cfc660cb3f519ae278 not found: ID does not exist" containerID="2dd461997d2bb59b6fc9a814fc1a5181d2de0f7b72cbd7cfc660cb3f519ae278"
Sep 16 16:57:07.777260 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.777194    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd461997d2bb59b6fc9a814fc1a5181d2de0f7b72cbd7cfc660cb3f519ae278"} err="failed to get container status \"2dd461997d2bb59b6fc9a814fc1a5181d2de0f7b72cbd7cfc660cb3f519ae278\": rpc error: code = NotFound desc = could not find container \"2dd461997d2bb59b6fc9a814fc1a5181d2de0f7b72cbd7cfc660cb3f519ae278\": container with ID starting with 2dd461997d2bb59b6fc9a814fc1a5181d2de0f7b72cbd7cfc660cb3f519ae278 not found: ID does not exist"
Sep 16 16:57:07.777260 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.777212    2574 scope.go:117] "RemoveContainer" containerID="9d370ca142a5b336527ee59867e1bf2520c7d261e5eaa11627185241331bfec2"
Sep 16 16:57:07.777492 ip-10-0-143-143 kubenswrapper[2574]: E0916 16:57:07.777461    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d370ca142a5b336527ee59867e1bf2520c7d261e5eaa11627185241331bfec2\": container with ID starting with 9d370ca142a5b336527ee59867e1bf2520c7d261e5eaa11627185241331bfec2 not found: ID does not exist" containerID="9d370ca142a5b336527ee59867e1bf2520c7d261e5eaa11627185241331bfec2"
Sep 16 16:57:07.777566 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.777501    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d370ca142a5b336527ee59867e1bf2520c7d261e5eaa11627185241331bfec2"} err="failed to get container status \"9d370ca142a5b336527ee59867e1bf2520c7d261e5eaa11627185241331bfec2\": rpc error: code = NotFound desc = could not find container \"9d370ca142a5b336527ee59867e1bf2520c7d261e5eaa11627185241331bfec2\": container with ID starting with 9d370ca142a5b336527ee59867e1bf2520c7d261e5eaa11627185241331bfec2 not found: ID does not exist"
Sep 16 16:57:07.845746 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.845712    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46fcb879-e90d-4b88-8ca8-51d040dfaf17-cabundle-cert\") pod \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") "
Sep 16 16:57:07.845912 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.845770    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-model-cache\") pod \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") "
Sep 16 16:57:07.845912 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.845812    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llkp6\" (UniqueName: \"kubernetes.io/projected/46fcb879-e90d-4b88-8ca8-51d040dfaf17-kube-api-access-llkp6\") pod \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") "
Sep 16 16:57:07.845912 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.845837    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-dshm\") pod \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") "
Sep 16 16:57:07.845912 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.845882    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46fcb879-e90d-4b88-8ca8-51d040dfaf17-tls-certs\") pod \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") "
Sep 16 16:57:07.846113 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.845912    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-tmp-dir\") pod \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") "
Sep 16 16:57:07.846113 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.845947    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-kserve-provision-location\") pod \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") "
Sep 16 16:57:07.846113 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.846001    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-home\") pod \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\" (UID: \"46fcb879-e90d-4b88-8ca8-51d040dfaf17\") "
Sep 16 16:57:07.846113 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.846084    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-model-cache" (OuterVolumeSpecName: "model-cache") pod "46fcb879-e90d-4b88-8ca8-51d040dfaf17" (UID: "46fcb879-e90d-4b88-8ca8-51d040dfaf17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:57:07.846325 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.846118    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fcb879-e90d-4b88-8ca8-51d040dfaf17-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "46fcb879-e90d-4b88-8ca8-51d040dfaf17" (UID: "46fcb879-e90d-4b88-8ca8-51d040dfaf17"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:57:07.846382 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.846321    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46fcb879-e90d-4b88-8ca8-51d040dfaf17-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:57:07.846382 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.846339    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:57:07.846481 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.846397    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-home" (OuterVolumeSpecName: "home") pod "46fcb879-e90d-4b88-8ca8-51d040dfaf17" (UID: "46fcb879-e90d-4b88-8ca8-51d040dfaf17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:57:07.848244 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.848212    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-dshm" (OuterVolumeSpecName: "dshm") pod "46fcb879-e90d-4b88-8ca8-51d040dfaf17" (UID: "46fcb879-e90d-4b88-8ca8-51d040dfaf17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:57:07.848377 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.848253    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fcb879-e90d-4b88-8ca8-51d040dfaf17-kube-api-access-llkp6" (OuterVolumeSpecName: "kube-api-access-llkp6") pod "46fcb879-e90d-4b88-8ca8-51d040dfaf17" (UID: "46fcb879-e90d-4b88-8ca8-51d040dfaf17"). InnerVolumeSpecName "kube-api-access-llkp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:57:07.848377 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.848303    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46fcb879-e90d-4b88-8ca8-51d040dfaf17-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46fcb879-e90d-4b88-8ca8-51d040dfaf17" (UID: "46fcb879-e90d-4b88-8ca8-51d040dfaf17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:57:07.861982 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.861929    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "46fcb879-e90d-4b88-8ca8-51d040dfaf17" (UID: "46fcb879-e90d-4b88-8ca8-51d040dfaf17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:57:07.877025 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.876981    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "46fcb879-e90d-4b88-8ca8-51d040dfaf17" (UID: "46fcb879-e90d-4b88-8ca8-51d040dfaf17"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:57:07.947880 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.947834    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llkp6\" (UniqueName: \"kubernetes.io/projected/46fcb879-e90d-4b88-8ca8-51d040dfaf17-kube-api-access-llkp6\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:57:07.947880 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.947872    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:57:07.947880 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.947883    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46fcb879-e90d-4b88-8ca8-51d040dfaf17-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:57:07.948319 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.947894    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:57:07.948319 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.947903    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:57:07.948319 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:07.947911    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46fcb879-e90d-4b88-8ca8-51d040dfaf17-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:57:08.077385 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:08.077287    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"]
Sep 16 16:57:08.082388 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:08.082360    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj"]
Sep 16 16:57:08.616810 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:08.616763    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cfffb9dc7-x8ctj" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Sep 16 16:57:09.994017 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:09.993977    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" path="/var/lib/kubelet/pods/46fcb879-e90d-4b88-8ca8-51d040dfaf17/volumes"
Sep 16 16:57:15.656537 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.656486    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"]
Sep 16 16:57:15.656933 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.656863    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerName="storage-initializer"
Sep 16 16:57:15.656933 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.656874    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerName="storage-initializer"
Sep 16 16:57:15.656933 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.656883    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerName="main"
Sep 16 16:57:15.656933 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.656889    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerName="main"
Sep 16 16:57:15.657068 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.656959    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="46fcb879-e90d-4b88-8ca8-51d040dfaf17" containerName="main"
Sep 16 16:57:15.661669 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.661651    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"
Sep 16 16:57:15.664240 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.664216    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 16 16:57:15.664405 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.664385    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 16 16:57:15.664498 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.664413    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-kv7d6\""
Sep 16 16:57:15.664498 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.664418    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 16 16:57:15.666018 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.665995    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"]
Sep 16 16:57:15.817953 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.817915    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ht55\" (UniqueName: \"kubernetes.io/projected/f2c0b132-dd9f-4849-a11f-45937142aaa9-kube-api-access-2ht55\") pod \"e2e-pvc-lora-download-mvvbd\" (UID: \"f2c0b132-dd9f-4849-a11f-45937142aaa9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"
Sep 16 16:57:15.918600 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.918485    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ht55\" (UniqueName: \"kubernetes.io/projected/f2c0b132-dd9f-4849-a11f-45937142aaa9-kube-api-access-2ht55\") pod \"e2e-pvc-lora-download-mvvbd\" (UID: \"f2c0b132-dd9f-4849-a11f-45937142aaa9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"
Sep 16 16:57:15.926914 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:15.926883    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ht55\" (UniqueName: \"kubernetes.io/projected/f2c0b132-dd9f-4849-a11f-45937142aaa9-kube-api-access-2ht55\") pod \"e2e-pvc-lora-download-mvvbd\" (UID: \"f2c0b132-dd9f-4849-a11f-45937142aaa9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"
Sep 16 16:57:18.043237 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:18.043191    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"e2e-pvc-lora-download-mvvbd\" (UID: \"f2c0b132-dd9f-4849-a11f-45937142aaa9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"
Sep 16 16:57:18.048838 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:18.048813    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"e2e-pvc-lora-download-mvvbd\" (UID: \"f2c0b132-dd9f-4849-a11f-45937142aaa9\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"
Sep 16 16:57:18.144438 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:18.144399    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"e2e-pvc-lora-download-mvvbd\" (UID: \"f2c0b132-dd9f-4849-a11f-45937142aaa9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"
Sep 16 16:57:18.144629 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:18.144571    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"e2e-pvc-lora-download-mvvbd\" (UID: \"f2c0b132-dd9f-4849-a11f-45937142aaa9\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"
Sep 16 16:57:18.151110 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:18.151089    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"e2e-pvc-lora-download-mvvbd\" (UID: \"f2c0b132-dd9f-4849-a11f-45937142aaa9\") DevicePath \"csi-eca16c938a41ddabd891d0f8fb3cbc6f413d93d121f1ba259653bf2c4e91aba9\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"
Sep 16 16:57:18.394950 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:18.394919    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"e2e-pvc-lora-download-mvvbd\" (UID: \"f2c0b132-dd9f-4849-a11f-45937142aaa9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/5b5d837e2841988b25f2d4175a7d6b77fda3dec62d65379cf01e4f015085a022/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"
Sep 16 16:57:18.407381 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:18.407346    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"e2e-pvc-lora-download-mvvbd\" (UID: \"f2c0b132-dd9f-4849-a11f-45937142aaa9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"
Sep 16 16:57:18.671330 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:18.671250    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"
Sep 16 16:57:18.940322 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:18.940295    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"]
Sep 16 16:57:18.942710 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:57:18.942685    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2c0b132_dd9f_4849_a11f_45937142aaa9.slice/crio-e30ae4098e65205d36fbc6b270f878290570288ed192649b4a90c79292653e23 WatchSource:0}: Error finding container e30ae4098e65205d36fbc6b270f878290570288ed192649b4a90c79292653e23: Status 404 returned error can't find the container with id e30ae4098e65205d36fbc6b270f878290570288ed192649b4a90c79292653e23
Sep 16 16:57:19.823084 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:19.823041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd" event={"ID":"f2c0b132-dd9f-4849-a11f-45937142aaa9","Type":"ContainerStarted","Data":"19d966757b73753584634d59378df974e03c6cf7d8fca06b679563719aa053a9"}
Sep 16 16:57:19.823619 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:19.823099    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd" event={"ID":"f2c0b132-dd9f-4849-a11f-45937142aaa9","Type":"ContainerStarted","Data":"e30ae4098e65205d36fbc6b270f878290570288ed192649b4a90c79292653e23"}
Sep 16 16:57:19.841358 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:19.841284    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd" podStartSLOduration=7.841261001 podStartE2EDuration="7.841261001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 16:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:57:19.839546556 +0000 UTC m=+4036.359169334" watchObservedRunningTime="2026-09-16 16:57:19.841261001 +0000 UTC m=+4036.360883779"
Sep 16 16:57:21.833939 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:21.833856    2574 generic.go:356] "Generic (PLEG): container finished" podID="f2c0b132-dd9f-4849-a11f-45937142aaa9" containerID="19d966757b73753584634d59378df974e03c6cf7d8fca06b679563719aa053a9" exitCode=0
Sep 16 16:57:21.834338 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:21.833929    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd" event={"ID":"f2c0b132-dd9f-4849-a11f-45937142aaa9","Type":"ContainerDied","Data":"19d966757b73753584634d59378df974e03c6cf7d8fca06b679563719aa053a9"}
Sep 16 16:57:22.966782 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:22.966757    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"
Sep 16 16:57:23.087787 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:23.087753    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ht55\" (UniqueName: \"kubernetes.io/projected/f2c0b132-dd9f-4849-a11f-45937142aaa9-kube-api-access-2ht55\") pod \"f2c0b132-dd9f-4849-a11f-45937142aaa9\" (UID: \"f2c0b132-dd9f-4849-a11f-45937142aaa9\") "
Sep 16 16:57:23.087973 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:23.087934    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"f2c0b132-dd9f-4849-a11f-45937142aaa9\" (UID: \"f2c0b132-dd9f-4849-a11f-45937142aaa9\") "
Sep 16 16:57:23.089993 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:23.089934    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c0b132-dd9f-4849-a11f-45937142aaa9-kube-api-access-2ht55" (OuterVolumeSpecName: "kube-api-access-2ht55") pod "f2c0b132-dd9f-4849-a11f-45937142aaa9" (UID: "f2c0b132-dd9f-4849-a11f-45937142aaa9"). InnerVolumeSpecName "kube-api-access-2ht55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:57:23.092089 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:23.092066    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59" (OuterVolumeSpecName: "model-storage") pod "f2c0b132-dd9f-4849-a11f-45937142aaa9" (UID: "f2c0b132-dd9f-4849-a11f-45937142aaa9"). InnerVolumeSpecName "pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 16:57:23.189579 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:23.189509    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ht55\" (UniqueName: \"kubernetes.io/projected/f2c0b132-dd9f-4849-a11f-45937142aaa9-kube-api-access-2ht55\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 16:57:23.189740 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:23.189597    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") on node \"ip-10-0-143-143.ec2.internal\" "
Sep 16 16:57:23.844154 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:23.844121    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"
Sep 16 16:57:23.844328 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:23.844124    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd" event={"ID":"f2c0b132-dd9f-4849-a11f-45937142aaa9","Type":"ContainerDied","Data":"e30ae4098e65205d36fbc6b270f878290570288ed192649b4a90c79292653e23"}
Sep 16 16:57:23.844328 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:23.844247    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e30ae4098e65205d36fbc6b270f878290570288ed192649b4a90c79292653e23"
Sep 16 16:57:24.207849 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:24.207826    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59") on node "ip-10-0-143-143.ec2.internal"
Sep 16 16:57:24.298416 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:24.298364    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"csi-eca16c938a41ddabd891d0f8fb3cbc6f413d93d121f1ba259653bf2c4e91aba9\""
Sep 16 16:57:36.754795 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:36.754719    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"
Sep 16 16:57:49.589236 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.589201    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"]
Sep 16 16:57:49.589751 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.589593    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2c0b132-dd9f-4849-a11f-45937142aaa9" containerName="storage-initializer"
Sep 16 16:57:49.589751 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.589605    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2c0b132-dd9f-4849-a11f-45937142aaa9" containerName="storage-initializer"
Sep 16 16:57:49.589751 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.589676    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2c0b132-dd9f-4849-a11f-45937142aaa9" containerName="storage-initializer"
Sep 16 16:57:49.593207 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.593178    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.595896 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.595816    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 16 16:57:49.595896 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.595871    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 16:57:49.596087 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.595900    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 16 16:57:49.596087 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.595812    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 16 16:57:49.596087 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.596022    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-kv7d6\""
Sep 16 16:57:49.596292 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.596273    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 16 16:57:49.604891 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.604870    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"]
Sep 16 16:57:49.726038 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.726003    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64772e05-e50d-4f42-976c-efbbb7ecbca0-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.726219 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.726045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.726219 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.726113    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz5vw\" (UniqueName: \"kubernetes.io/projected/64772e05-e50d-4f42-976c-efbbb7ecbca0-kube-api-access-wz5vw\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.726219 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.726164    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-dshm\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.726219 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.726204    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-home\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.726430 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.726251    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.726430 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.726287    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64772e05-e50d-4f42-976c-efbbb7ecbca0-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.726430 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.726310    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.827492 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.827454    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz5vw\" (UniqueName: \"kubernetes.io/projected/64772e05-e50d-4f42-976c-efbbb7ecbca0-kube-api-access-wz5vw\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.827663 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.827497    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-dshm\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.827663 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.827635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-home\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.827767 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.827673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.827767 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.827708    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64772e05-e50d-4f42-976c-efbbb7ecbca0-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.827767 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.827731    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.827918 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.827808    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64772e05-e50d-4f42-976c-efbbb7ecbca0-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.827918 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.827846    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.828083 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.828058    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-home\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.828139 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.828078    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.828177 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.828146    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.828301 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.828277    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.828441 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.828423    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64772e05-e50d-4f42-976c-efbbb7ecbca0-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.829819 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.829799    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-dshm\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.830306 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.830286    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64772e05-e50d-4f42-976c-efbbb7ecbca0-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:49.834533 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:49.834497    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz5vw\" (UniqueName: \"kubernetes.io/projected/64772e05-e50d-4f42-976c-efbbb7ecbca0-kube-api-access-wz5vw\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:58.909946 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:58.909907    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:58.915365 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:58.915343    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:59.010345 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:59.010320    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:59.010494 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:59.010437    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:59.012679 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:59.012660    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") DevicePath \"csi-eca16c938a41ddabd891d0f8fb3cbc6f413d93d121f1ba259653bf2c4e91aba9\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:59.099761 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:59.099738    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/5b5d837e2841988b25f2d4175a7d6b77fda3dec62d65379cf01e4f015085a022/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:59.118200 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:59.118177    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:59.205047 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:59.204991    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:57:59.478242 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:57:59.478214    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"]
Sep 16 16:57:59.480904 ip-10-0-143-143 kubenswrapper[2574]: W0916 16:57:59.480872    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64772e05_e50d_4f42_976c_efbbb7ecbca0.slice/crio-069b79897a4421e1d09f4336d59faff0dc0e4e82801813c78ac03d938b0a78a7 WatchSource:0}: Error finding container 069b79897a4421e1d09f4336d59faff0dc0e4e82801813c78ac03d938b0a78a7: Status 404 returned error can't find the container with id 069b79897a4421e1d09f4336d59faff0dc0e4e82801813c78ac03d938b0a78a7
Sep 16 16:58:00.016845 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:58:00.016803    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" event={"ID":"64772e05-e50d-4f42-976c-efbbb7ecbca0","Type":"ContainerStarted","Data":"5445fb217f82b940b7d10bc2f22ff444a6cac645fbe1a1f773c2eab2cd5562ae"}
Sep 16 16:58:00.017285 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:58:00.016857    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" event={"ID":"64772e05-e50d-4f42-976c-efbbb7ecbca0","Type":"ContainerStarted","Data":"069b79897a4421e1d09f4336d59faff0dc0e4e82801813c78ac03d938b0a78a7"}
Sep 16 16:58:02.031665 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:58:02.031570    2574 generic.go:356] "Generic (PLEG): container finished" podID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerID="5445fb217f82b940b7d10bc2f22ff444a6cac645fbe1a1f773c2eab2cd5562ae" exitCode=0
Sep 16 16:58:02.032102 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:58:02.031645    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" event={"ID":"64772e05-e50d-4f42-976c-efbbb7ecbca0","Type":"ContainerDied","Data":"5445fb217f82b940b7d10bc2f22ff444a6cac645fbe1a1f773c2eab2cd5562ae"}
Sep 16 16:58:03.039229 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:58:03.039192    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" event={"ID":"64772e05-e50d-4f42-976c-efbbb7ecbca0","Type":"ContainerStarted","Data":"ad81b4ca1a4ff25f7fce11a3b88cebee6e54996217b195113047588cac0ed175"}
Sep 16 16:58:03.063234 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:58:03.063176    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" podStartSLOduration=11.931759984 podStartE2EDuration="14.06315846s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.131398476s" isStatefulPod=true podCreationTimestamp="2026-09-16 16:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:58:03.061934527 +0000 UTC m=+4079.581557303" watchObservedRunningTime="2026-09-16 16:58:03.06315846 +0000 UTC m=+4079.582781232"
Sep 16 16:58:03.205662 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:58:03.205623    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:58:09.205535 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:58:09.205475    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:58:09.207245 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:58:09.207217    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" podUID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 16:58:19.205536 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:58:19.205466    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" podUID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 16:58:29.205652 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:58:29.205614    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" podUID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 16:58:39.206095 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:58:39.206047    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" podUID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 16:58:49.205672 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:58:49.205627    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" podUID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 16:58:59.206217 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:58:59.206122    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" podUID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 16:59:09.205729 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:59:09.205683    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" podUID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 16:59:19.214983 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:59:19.214951    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:59:19.222359 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:59:19.222330    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 16:59:44.086300 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:59:44.086248    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb_64772e05-e50d-4f42-976c-efbbb7ecbca0/storage-initializer/0.log"
Sep 16 16:59:44.094795 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:59:44.094770    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb_64772e05-e50d-4f42-976c-efbbb7ecbca0/main/0.log"
Sep 16 16:59:47.309379 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:59:47.309340    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"]
Sep 16 16:59:54.116757 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:59:54.116722    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"]
Sep 16 16:59:56.754818 ip-10-0-143-143 kubenswrapper[2574]: I0916 16:59:56.754773    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx" podUID="fa87d418-dc20-4e57-874c-84ee472be47d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:00:02.332334 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:02.332286    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" podUID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerName="main" containerID="cri-o://ad81b4ca1a4ff25f7fce11a3b88cebee6e54996217b195113047588cac0ed175" gracePeriod=45
Sep 16 17:00:02.988847 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:02.988821    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 17:00:03.093243 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.093151    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-model-cache\") pod \"64772e05-e50d-4f42-976c-efbbb7ecbca0\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") "
Sep 16 17:00:03.093400 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.093283    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") pod \"64772e05-e50d-4f42-976c-efbbb7ecbca0\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") "
Sep 16 17:00:03.093400 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.093304    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-dshm\") pod \"64772e05-e50d-4f42-976c-efbbb7ecbca0\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") "
Sep 16 17:00:03.093400 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.093325    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-home\") pod \"64772e05-e50d-4f42-976c-efbbb7ecbca0\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") "
Sep 16 17:00:03.093400 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.093361    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz5vw\" (UniqueName: \"kubernetes.io/projected/64772e05-e50d-4f42-976c-efbbb7ecbca0-kube-api-access-wz5vw\") pod \"64772e05-e50d-4f42-976c-efbbb7ecbca0\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") "
Sep 16 17:00:03.093400 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.093395    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64772e05-e50d-4f42-976c-efbbb7ecbca0-cabundle-cert\") pod \"64772e05-e50d-4f42-976c-efbbb7ecbca0\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") "
Sep 16 17:00:03.093710 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.093440    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-tmp-dir\") pod \"64772e05-e50d-4f42-976c-efbbb7ecbca0\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") "
Sep 16 17:00:03.093710 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.093449    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-model-cache" (OuterVolumeSpecName: "model-cache") pod "64772e05-e50d-4f42-976c-efbbb7ecbca0" (UID: "64772e05-e50d-4f42-976c-efbbb7ecbca0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:03.093710 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.093472    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64772e05-e50d-4f42-976c-efbbb7ecbca0-tls-certs\") pod \"64772e05-e50d-4f42-976c-efbbb7ecbca0\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") "
Sep 16 17:00:03.093710 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.093507    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-kserve-provision-location\") pod \"64772e05-e50d-4f42-976c-efbbb7ecbca0\" (UID: \"64772e05-e50d-4f42-976c-efbbb7ecbca0\") "
Sep 16 17:00:03.093899 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.093798    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-home" (OuterVolumeSpecName: "home") pod "64772e05-e50d-4f42-976c-efbbb7ecbca0" (UID: "64772e05-e50d-4f42-976c-efbbb7ecbca0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:03.093899 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.093808    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:00:03.093899 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.093846    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64772e05-e50d-4f42-976c-efbbb7ecbca0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64772e05-e50d-4f42-976c-efbbb7ecbca0" (UID: "64772e05-e50d-4f42-976c-efbbb7ecbca0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:00:03.095879 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.095846    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-dshm" (OuterVolumeSpecName: "dshm") pod "64772e05-e50d-4f42-976c-efbbb7ecbca0" (UID: "64772e05-e50d-4f42-976c-efbbb7ecbca0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:03.096086 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.096033    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64772e05-e50d-4f42-976c-efbbb7ecbca0-kube-api-access-wz5vw" (OuterVolumeSpecName: "kube-api-access-wz5vw") pod "64772e05-e50d-4f42-976c-efbbb7ecbca0" (UID: "64772e05-e50d-4f42-976c-efbbb7ecbca0"). InnerVolumeSpecName "kube-api-access-wz5vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:00:03.096164 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.096085    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64772e05-e50d-4f42-976c-efbbb7ecbca0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64772e05-e50d-4f42-976c-efbbb7ecbca0" (UID: "64772e05-e50d-4f42-976c-efbbb7ecbca0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:00:03.098807 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.098779    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "64772e05-e50d-4f42-976c-efbbb7ecbca0" (UID: "64772e05-e50d-4f42-976c-efbbb7ecbca0"). InnerVolumeSpecName "pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 17:00:03.106465 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.106439    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64772e05-e50d-4f42-976c-efbbb7ecbca0" (UID: "64772e05-e50d-4f42-976c-efbbb7ecbca0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:03.118178 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.118151    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64772e05-e50d-4f42-976c-efbbb7ecbca0" (UID: "64772e05-e50d-4f42-976c-efbbb7ecbca0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:00:03.194818 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.194788    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64772e05-e50d-4f42-976c-efbbb7ecbca0-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:00:03.194818 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.194818    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:00:03.194983 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.194851    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") on node \"ip-10-0-143-143.ec2.internal\" "
Sep 16 17:00:03.194983 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.194863    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:00:03.194983 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.194871    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:00:03.194983 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.194879    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz5vw\" (UniqueName: \"kubernetes.io/projected/64772e05-e50d-4f42-976c-efbbb7ecbca0-kube-api-access-wz5vw\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:00:03.194983 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.194890    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64772e05-e50d-4f42-976c-efbbb7ecbca0-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:00:03.194983 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.194898    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64772e05-e50d-4f42-976c-efbbb7ecbca0-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:00:03.204823 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.204804    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59") on node "ip-10-0-143-143.ec2.internal"
Sep 16 17:00:03.296030 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.295992    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-9fd9d8e1-01e1-4c92-b72e-f6465a24cfff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b193f91d0399dc59\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"csi-eca16c938a41ddabd891d0f8fb3cbc6f413d93d121f1ba259653bf2c4e91aba9\""
Sep 16 17:00:03.665287 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.665250    2574 generic.go:356] "Generic (PLEG): container finished" podID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerID="ad81b4ca1a4ff25f7fce11a3b88cebee6e54996217b195113047588cac0ed175" exitCode=0
Sep 16 17:00:03.665773 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.665308    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"
Sep 16 17:00:03.665773 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.665328    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" event={"ID":"64772e05-e50d-4f42-976c-efbbb7ecbca0","Type":"ContainerDied","Data":"ad81b4ca1a4ff25f7fce11a3b88cebee6e54996217b195113047588cac0ed175"}
Sep 16 17:00:03.665773 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.665385    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb" event={"ID":"64772e05-e50d-4f42-976c-efbbb7ecbca0","Type":"ContainerDied","Data":"069b79897a4421e1d09f4336d59faff0dc0e4e82801813c78ac03d938b0a78a7"}
Sep 16 17:00:03.665773 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.665414    2574 scope.go:117] "RemoveContainer" containerID="ad81b4ca1a4ff25f7fce11a3b88cebee6e54996217b195113047588cac0ed175"
Sep 16 17:00:03.675103 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.675070    2574 scope.go:117] "RemoveContainer" containerID="5445fb217f82b940b7d10bc2f22ff444a6cac645fbe1a1f773c2eab2cd5562ae"
Sep 16 17:00:03.684583 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.684212    2574 scope.go:117] "RemoveContainer" containerID="ad81b4ca1a4ff25f7fce11a3b88cebee6e54996217b195113047588cac0ed175"
Sep 16 17:00:03.684583 ip-10-0-143-143 kubenswrapper[2574]: E0916 17:00:03.684548    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad81b4ca1a4ff25f7fce11a3b88cebee6e54996217b195113047588cac0ed175\": container with ID starting with ad81b4ca1a4ff25f7fce11a3b88cebee6e54996217b195113047588cac0ed175 not found: ID does not exist" containerID="ad81b4ca1a4ff25f7fce11a3b88cebee6e54996217b195113047588cac0ed175"
Sep 16 17:00:03.684756 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.684577    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad81b4ca1a4ff25f7fce11a3b88cebee6e54996217b195113047588cac0ed175"} err="failed to get container status \"ad81b4ca1a4ff25f7fce11a3b88cebee6e54996217b195113047588cac0ed175\": rpc error: code = NotFound desc = could not find container \"ad81b4ca1a4ff25f7fce11a3b88cebee6e54996217b195113047588cac0ed175\": container with ID starting with ad81b4ca1a4ff25f7fce11a3b88cebee6e54996217b195113047588cac0ed175 not found: ID does not exist"
Sep 16 17:00:03.684756 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.684602    2574 scope.go:117] "RemoveContainer" containerID="5445fb217f82b940b7d10bc2f22ff444a6cac645fbe1a1f773c2eab2cd5562ae"
Sep 16 17:00:03.685229 ip-10-0-143-143 kubenswrapper[2574]: E0916 17:00:03.685211    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5445fb217f82b940b7d10bc2f22ff444a6cac645fbe1a1f773c2eab2cd5562ae\": container with ID starting with 5445fb217f82b940b7d10bc2f22ff444a6cac645fbe1a1f773c2eab2cd5562ae not found: ID does not exist" containerID="5445fb217f82b940b7d10bc2f22ff444a6cac645fbe1a1f773c2eab2cd5562ae"
Sep 16 17:00:03.685304 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.685235    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5445fb217f82b940b7d10bc2f22ff444a6cac645fbe1a1f773c2eab2cd5562ae"} err="failed to get container status \"5445fb217f82b940b7d10bc2f22ff444a6cac645fbe1a1f773c2eab2cd5562ae\": rpc error: code = NotFound desc = could not find container \"5445fb217f82b940b7d10bc2f22ff444a6cac645fbe1a1f773c2eab2cd5562ae\": container with ID starting with 5445fb217f82b940b7d10bc2f22ff444a6cac645fbe1a1f773c2eab2cd5562ae not found: ID does not exist"
Sep 16 17:00:03.688011 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.687988    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"]
Sep 16 17:00:03.693470 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.693449    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5d4957cdd8-9ntpb"]
Sep 16 17:00:03.994999 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:03.994926    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64772e05-e50d-4f42-976c-efbbb7ecbca0" path="/var/lib/kubelet/pods/64772e05-e50d-4f42-976c-efbbb7ecbca0/volumes"
Sep 16 17:00:06.754907 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:06.754856    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx" podUID="fa87d418-dc20-4e57-874c-84ee472be47d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:00:08.588250 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.588211    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"]
Sep 16 17:00:08.588677 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.588606    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerName="storage-initializer"
Sep 16 17:00:08.588677 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.588618    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerName="storage-initializer"
Sep 16 17:00:08.588677 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.588627    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerName="main"
Sep 16 17:00:08.588677 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.588632    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerName="main"
Sep 16 17:00:08.588833 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.588703    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="64772e05-e50d-4f42-976c-efbbb7ecbca0" containerName="main"
Sep 16 17:00:08.593487 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.593469    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"
Sep 16 17:00:08.599666 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.599638    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 16 17:00:08.599865 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.599767    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 16 17:00:08.600202 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.600109    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-rfdsk\""
Sep 16 17:00:08.600378 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.600361    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 16 17:00:08.604482 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.604456    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"]
Sep 16 17:00:08.647180 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.647155    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6kg6\" (UniqueName: \"kubernetes.io/projected/be6ef111-8d95-4b29-8719-c0b9574db73c-kube-api-access-z6kg6\") pod \"e2e-pvc-lora-download-nrjp8\" (UID: \"be6ef111-8d95-4b29-8719-c0b9574db73c\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"
Sep 16 17:00:08.747903 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.747866    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6kg6\" (UniqueName: \"kubernetes.io/projected/be6ef111-8d95-4b29-8719-c0b9574db73c-kube-api-access-z6kg6\") pod \"e2e-pvc-lora-download-nrjp8\" (UID: \"be6ef111-8d95-4b29-8719-c0b9574db73c\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"
Sep 16 17:00:08.757585 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:08.757557    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6kg6\" (UniqueName: \"kubernetes.io/projected/be6ef111-8d95-4b29-8719-c0b9574db73c-kube-api-access-z6kg6\") pod \"e2e-pvc-lora-download-nrjp8\" (UID: \"be6ef111-8d95-4b29-8719-c0b9574db73c\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"
Sep 16 17:00:09.136300 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:09.136263    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx" podUID="fa87d418-dc20-4e57-874c-84ee472be47d" containerName="main" containerID="cri-o://148220f06e3598a3309775358a719f4edb7691b5b3070b47ceccdce0a976f3a3" gracePeriod=45
Sep 16 17:00:10.280791 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:10.280749    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"]
Sep 16 17:00:10.282955 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:10.282933    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mvvbd"]
Sep 16 17:00:11.994699 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:11.994662    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2c0b132-dd9f-4849-a11f-45937142aaa9" path="/var/lib/kubelet/pods/f2c0b132-dd9f-4849-a11f-45937142aaa9/volumes"
Sep 16 17:00:12.287611 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:12.287500    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"e2e-pvc-lora-download-nrjp8\" (UID: \"be6ef111-8d95-4b29-8719-c0b9574db73c\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"
Sep 16 17:00:12.292279 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:12.292251    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"e2e-pvc-lora-download-nrjp8\" (UID: \"be6ef111-8d95-4b29-8719-c0b9574db73c\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"
Sep 16 17:00:12.388303 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:12.388264    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"e2e-pvc-lora-download-nrjp8\" (UID: \"be6ef111-8d95-4b29-8719-c0b9574db73c\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"
Sep 16 17:00:12.388480 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:12.388415    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"e2e-pvc-lora-download-nrjp8\" (UID: \"be6ef111-8d95-4b29-8719-c0b9574db73c\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"
Sep 16 17:00:12.390923 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:12.390902    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"e2e-pvc-lora-download-nrjp8\" (UID: \"be6ef111-8d95-4b29-8719-c0b9574db73c\") DevicePath \"csi-a323ce815e0b76d12bef5553a7f9e72a8bc67a158e154da7d151e5e4f9c2f678\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"
Sep 16 17:00:12.598205 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:12.598170    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"e2e-pvc-lora-download-nrjp8\" (UID: \"be6ef111-8d95-4b29-8719-c0b9574db73c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9e751b0d9c270464a55c9c1a9c7fad2b709bde7e1c07c4c747f57d8ce0e87f02/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"
Sep 16 17:00:12.611813 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:12.611784    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"e2e-pvc-lora-download-nrjp8\" (UID: \"be6ef111-8d95-4b29-8719-c0b9574db73c\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"
Sep 16 17:00:12.802333 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:12.802294    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"
Sep 16 17:00:13.070296 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:13.070269    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"]
Sep 16 17:00:13.072276 ip-10-0-143-143 kubenswrapper[2574]: W0916 17:00:13.072253    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe6ef111_8d95_4b29_8719_c0b9574db73c.slice/crio-8ba3c065ce9489bc8225905eddf258c7e9afc06570a624f69bc06b2b22f6c0c9 WatchSource:0}: Error finding container 8ba3c065ce9489bc8225905eddf258c7e9afc06570a624f69bc06b2b22f6c0c9: Status 404 returned error can't find the container with id 8ba3c065ce9489bc8225905eddf258c7e9afc06570a624f69bc06b2b22f6c0c9
Sep 16 17:00:13.713194 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:13.713158    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8" event={"ID":"be6ef111-8d95-4b29-8719-c0b9574db73c","Type":"ContainerStarted","Data":"bc260c6e77325bbc04ea8a5c44e0835b8fa6c735d13f2ed3be5212efe79488b9"}
Sep 16 17:00:13.713379 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:13.713205    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8" event={"ID":"be6ef111-8d95-4b29-8719-c0b9574db73c","Type":"ContainerStarted","Data":"8ba3c065ce9489bc8225905eddf258c7e9afc06570a624f69bc06b2b22f6c0c9"}
Sep 16 17:00:13.728764 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:13.728714    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8" podStartSLOduration=8.728696683 podStartE2EDuration="8.728696683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:00:13.727105053 +0000 UTC m=+4210.246727819" watchObservedRunningTime="2026-09-16 17:00:13.728696683 +0000 UTC m=+4210.248319459"
Sep 16 17:00:15.726941 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:15.726904    2574 generic.go:356] "Generic (PLEG): container finished" podID="be6ef111-8d95-4b29-8719-c0b9574db73c" containerID="bc260c6e77325bbc04ea8a5c44e0835b8fa6c735d13f2ed3be5212efe79488b9" exitCode=0
Sep 16 17:00:15.727345 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:15.726978    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8" event={"ID":"be6ef111-8d95-4b29-8719-c0b9574db73c","Type":"ContainerDied","Data":"bc260c6e77325bbc04ea8a5c44e0835b8fa6c735d13f2ed3be5212efe79488b9"}
Sep 16 17:00:16.755546 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:16.755493    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx" podUID="fa87d418-dc20-4e57-874c-84ee472be47d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:00:16.756030 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:16.755626    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"
Sep 16 17:00:16.871131 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:16.871098    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"
Sep 16 17:00:16.933140 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:16.933107    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") pod \"be6ef111-8d95-4b29-8719-c0b9574db73c\" (UID: \"be6ef111-8d95-4b29-8719-c0b9574db73c\") "
Sep 16 17:00:16.933334 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:16.933165    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6kg6\" (UniqueName: \"kubernetes.io/projected/be6ef111-8d95-4b29-8719-c0b9574db73c-kube-api-access-z6kg6\") pod \"be6ef111-8d95-4b29-8719-c0b9574db73c\" (UID: \"be6ef111-8d95-4b29-8719-c0b9574db73c\") "
Sep 16 17:00:16.935481 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:16.935452    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6ef111-8d95-4b29-8719-c0b9574db73c-kube-api-access-z6kg6" (OuterVolumeSpecName: "kube-api-access-z6kg6") pod "be6ef111-8d95-4b29-8719-c0b9574db73c" (UID: "be6ef111-8d95-4b29-8719-c0b9574db73c"). InnerVolumeSpecName "kube-api-access-z6kg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:00:16.935882 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:16.935862    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5" (OuterVolumeSpecName: "model-storage") pod "be6ef111-8d95-4b29-8719-c0b9574db73c" (UID: "be6ef111-8d95-4b29-8719-c0b9574db73c"). InnerVolumeSpecName "pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 17:00:17.034224 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:17.034118    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6kg6\" (UniqueName: \"kubernetes.io/projected/be6ef111-8d95-4b29-8719-c0b9574db73c-kube-api-access-z6kg6\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:00:17.034224 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:17.034171    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") on node \"ip-10-0-143-143.ec2.internal\" "
Sep 16 17:00:17.740424 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:17.740386    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8" event={"ID":"be6ef111-8d95-4b29-8719-c0b9574db73c","Type":"ContainerDied","Data":"8ba3c065ce9489bc8225905eddf258c7e9afc06570a624f69bc06b2b22f6c0c9"}
Sep 16 17:00:17.740424 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:17.740416    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"
Sep 16 17:00:17.740655 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:17.740429    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ba3c065ce9489bc8225905eddf258c7e9afc06570a624f69bc06b2b22f6c0c9"
Sep 16 17:00:18.004877 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:18.004856    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5") on node "ip-10-0-143-143.ec2.internal"
Sep 16 17:00:18.043349 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:18.043309    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-f6a115bc-fe2c-4d81-8231-1cd9cb25028e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0798b6fdf777dd2e5\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"csi-a323ce815e0b76d12bef5553a7f9e72a8bc67a158e154da7d151e5e4f9c2f678\""
Sep 16 17:00:26.754428 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:26.754350    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx" podUID="fa87d418-dc20-4e57-874c-84ee472be47d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:00:36.755234 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:36.755190    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx" podUID="fa87d418-dc20-4e57-874c-84ee472be47d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:00:39.376192 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.376168    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"
Sep 16 17:00:39.439656 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.439572    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa87d418-dc20-4e57-874c-84ee472be47d-tls-certs\") pod \"fa87d418-dc20-4e57-874c-84ee472be47d\" (UID: \"fa87d418-dc20-4e57-874c-84ee472be47d\") "
Sep 16 17:00:39.439793 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.439683    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrpcw\" (UniqueName: \"kubernetes.io/projected/fa87d418-dc20-4e57-874c-84ee472be47d-kube-api-access-zrpcw\") pod \"fa87d418-dc20-4e57-874c-84ee472be47d\" (UID: \"fa87d418-dc20-4e57-874c-84ee472be47d\") "
Sep 16 17:00:39.441501 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.441477    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa87d418-dc20-4e57-874c-84ee472be47d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa87d418-dc20-4e57-874c-84ee472be47d" (UID: "fa87d418-dc20-4e57-874c-84ee472be47d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:00:39.441634 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.441617    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa87d418-dc20-4e57-874c-84ee472be47d-kube-api-access-zrpcw" (OuterVolumeSpecName: "kube-api-access-zrpcw") pod "fa87d418-dc20-4e57-874c-84ee472be47d" (UID: "fa87d418-dc20-4e57-874c-84ee472be47d"). InnerVolumeSpecName "kube-api-access-zrpcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:00:39.540767 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.540726    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrpcw\" (UniqueName: \"kubernetes.io/projected/fa87d418-dc20-4e57-874c-84ee472be47d-kube-api-access-zrpcw\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:00:39.540767 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.540763    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa87d418-dc20-4e57-874c-84ee472be47d-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:00:39.845653 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.845563    2574 generic.go:356] "Generic (PLEG): container finished" podID="fa87d418-dc20-4e57-874c-84ee472be47d" containerID="148220f06e3598a3309775358a719f4edb7691b5b3070b47ceccdce0a976f3a3" exitCode=0
Sep 16 17:00:39.845653 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.845602    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"
Sep 16 17:00:39.845653 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.845642    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx" event={"ID":"fa87d418-dc20-4e57-874c-84ee472be47d","Type":"ContainerDied","Data":"148220f06e3598a3309775358a719f4edb7691b5b3070b47ceccdce0a976f3a3"}
Sep 16 17:00:39.845872 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.845690    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx" event={"ID":"fa87d418-dc20-4e57-874c-84ee472be47d","Type":"ContainerDied","Data":"ee55a2f0c6cccada991094ecbd84b3052402e45a8b0112510d15ec5e2dd9f5b8"}
Sep 16 17:00:39.845872 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.845710    2574 scope.go:117] "RemoveContainer" containerID="148220f06e3598a3309775358a719f4edb7691b5b3070b47ceccdce0a976f3a3"
Sep 16 17:00:39.855161 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.855144    2574 scope.go:117] "RemoveContainer" containerID="148220f06e3598a3309775358a719f4edb7691b5b3070b47ceccdce0a976f3a3"
Sep 16 17:00:39.855410 ip-10-0-143-143 kubenswrapper[2574]: E0916 17:00:39.855391    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"148220f06e3598a3309775358a719f4edb7691b5b3070b47ceccdce0a976f3a3\": container with ID starting with 148220f06e3598a3309775358a719f4edb7691b5b3070b47ceccdce0a976f3a3 not found: ID does not exist" containerID="148220f06e3598a3309775358a719f4edb7691b5b3070b47ceccdce0a976f3a3"
Sep 16 17:00:39.855457 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.855416    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"148220f06e3598a3309775358a719f4edb7691b5b3070b47ceccdce0a976f3a3"} err="failed to get container status \"148220f06e3598a3309775358a719f4edb7691b5b3070b47ceccdce0a976f3a3\": rpc error: code = NotFound desc = could not find container \"148220f06e3598a3309775358a719f4edb7691b5b3070b47ceccdce0a976f3a3\": container with ID starting with 148220f06e3598a3309775358a719f4edb7691b5b3070b47ceccdce0a976f3a3 not found: ID does not exist"
Sep 16 17:00:39.867127 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.867100    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"]
Sep 16 17:00:39.870869 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.870836    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6tnlx"]
Sep 16 17:00:39.994310 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:00:39.994274    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa87d418-dc20-4e57-874c-84ee472be47d" path="/var/lib/kubelet/pods/fa87d418-dc20-4e57-874c-84ee472be47d/volumes"
Sep 16 17:01:36.922036 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.921998    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"]
Sep 16 17:01:36.922665 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.922366    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa87d418-dc20-4e57-874c-84ee472be47d" containerName="main"
Sep 16 17:01:36.922665 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.922378    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa87d418-dc20-4e57-874c-84ee472be47d" containerName="main"
Sep 16 17:01:36.922665 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.922396    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be6ef111-8d95-4b29-8719-c0b9574db73c" containerName="storage-initializer"
Sep 16 17:01:36.922665 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.922402    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="be6ef111-8d95-4b29-8719-c0b9574db73c" containerName="storage-initializer"
Sep 16 17:01:36.922665 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.922459    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="be6ef111-8d95-4b29-8719-c0b9574db73c" containerName="storage-initializer"
Sep 16 17:01:36.922665 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.922467    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa87d418-dc20-4e57-874c-84ee472be47d" containerName="main"
Sep 16 17:01:36.925927 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.925906    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:36.928203 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.928174    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 17:01:36.928353 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.928250    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 17:01:36.928498 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.928475    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-67t67\""
Sep 16 17:01:36.928598 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.928493    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 17:01:36.928598 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.928478    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:01:36.928598 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.928580    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-wgkhj\""
Sep 16 17:01:36.928855 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.928839    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 17:01:36.941568 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:36.941541    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"]
Sep 16 17:01:37.039744 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.039704    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a180882-99e0-4672-80ba-00b5d9753ce1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.039918 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.039757    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-dshm\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.039918 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.039822    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.039918 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.039849    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a180882-99e0-4672-80ba-00b5d9753ce1-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.039918 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.039887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.040061 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.039926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-home\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.040061 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.039966    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.040061 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.039991    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqrp\" (UniqueName: \"kubernetes.io/projected/4a180882-99e0-4672-80ba-00b5d9753ce1-kube-api-access-tfqrp\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.140798 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.140761    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-dshm\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.140972 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.140813    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.140972 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.140837    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a180882-99e0-4672-80ba-00b5d9753ce1-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.140972 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.140885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.140972 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.140924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-home\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.140972 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.140956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.141209 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.141083    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqrp\" (UniqueName: \"kubernetes.io/projected/4a180882-99e0-4672-80ba-00b5d9753ce1-kube-api-access-tfqrp\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.141209 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.141185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a180882-99e0-4672-80ba-00b5d9753ce1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.141315 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.141268    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.141373 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.141329    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.141473 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.141452    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.141633 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.141486    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-home\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.141697 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.141656    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a180882-99e0-4672-80ba-00b5d9753ce1-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.143222 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.143201    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-dshm\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.143739 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.143718    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a180882-99e0-4672-80ba-00b5d9753ce1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.149435 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.149415    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqrp\" (UniqueName: \"kubernetes.io/projected/4a180882-99e0-4672-80ba-00b5d9753ce1-kube-api-access-tfqrp\") pod \"custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.235137 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.235048    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:37.512708 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.512667    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"]
Sep 16 17:01:37.513909 ip-10-0-143-143 kubenswrapper[2574]: W0916 17:01:37.513886    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a180882_99e0_4672_80ba_00b5d9753ce1.slice/crio-5b8c157f340b7f74d529fb7499b6ca5b787f0df22907a9f942d066912c4da9e1 WatchSource:0}: Error finding container 5b8c157f340b7f74d529fb7499b6ca5b787f0df22907a9f942d066912c4da9e1: Status 404 returned error can't find the container with id 5b8c157f340b7f74d529fb7499b6ca5b787f0df22907a9f942d066912c4da9e1
Sep 16 17:01:37.516088 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:37.516070    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:01:38.119995 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:38.119958    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" event={"ID":"4a180882-99e0-4672-80ba-00b5d9753ce1","Type":"ContainerStarted","Data":"5b8c157f340b7f74d529fb7499b6ca5b787f0df22907a9f942d066912c4da9e1"}
Sep 16 17:01:40.132297 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:40.132259    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" event={"ID":"4a180882-99e0-4672-80ba-00b5d9753ce1","Type":"ContainerStarted","Data":"9c4be8b19eca3382fcc5d9ebef4c2edc8f234bd89d0e86e8b477e9d7f44f4cbd"}
Sep 16 17:01:40.132721 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:40.132391    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:41.139021 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:41.138986    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" event={"ID":"4a180882-99e0-4672-80ba-00b5d9753ce1","Type":"ContainerStarted","Data":"7bd8ff93119f0a0b929b630025669da11be6d32a9420757c0749599085cddb72"}
Sep 16 17:01:43.152094 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:43.152060    2574 generic.go:356] "Generic (PLEG): container finished" podID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerID="7bd8ff93119f0a0b929b630025669da11be6d32a9420757c0749599085cddb72" exitCode=0
Sep 16 17:01:43.152480 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:43.152124    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" event={"ID":"4a180882-99e0-4672-80ba-00b5d9753ce1","Type":"ContainerDied","Data":"7bd8ff93119f0a0b929b630025669da11be6d32a9420757c0749599085cddb72"}
Sep 16 17:01:44.161407 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:44.161365    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" event={"ID":"4a180882-99e0-4672-80ba-00b5d9753ce1","Type":"ContainerStarted","Data":"515742277a6d636c3d618a84eabffb32485d7940754fd8188b29fd7cd76afe1f"}
Sep 16 17:01:44.183986 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:44.183924    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" podStartSLOduration=4.234635812 podStartE2EDuration="8.183904051s" totalImagesPullingTime="1.722441007s" totalInitContainerRuntime="2.226827232s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:01:44.18344198 +0000 UTC m=+4300.703064762" watchObservedRunningTime="2026-09-16 17:01:44.183904051 +0000 UTC m=+4300.703526829"
Sep 16 17:01:47.235440 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:47.235356    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:47.235440 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:47.235417    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:01:47.236731 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:47.236703    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 17:01:57.236113 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:57.236014    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 17:01:57.253431 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:01:57.253402    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:02:07.236027 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:02:07.235916    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 17:02:17.235844 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:02:17.235799    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 17:02:27.236231 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:02:27.236180    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 17:02:37.236431 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:02:37.236302    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 17:02:47.236383 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:02:47.236265    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 17:02:48.884301 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:02:48.884262    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"]
Sep 16 17:02:48.887901 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:02:48.887875    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nrjp8"]
Sep 16 17:02:49.994926 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:02:49.994891    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be6ef111-8d95-4b29-8719-c0b9574db73c" path="/var/lib/kubelet/pods/be6ef111-8d95-4b29-8719-c0b9574db73c/volumes"
Sep 16 17:02:57.235867 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:02:57.235818    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 17:03:01.430144 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.430107    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"]
Sep 16 17:03:01.434003 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.433967    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"
Sep 16 17:03:01.437107 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.437083    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-tdqq7\""
Sep 16 17:03:01.437107 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.437102    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 16 17:03:01.437593 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.437086    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-vzmf9\""
Sep 16 17:03:01.437593 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.437153    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 16 17:03:01.437593 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.437093    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 17:03:01.440167 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.440148    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"]
Sep 16 17:03:01.531939 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.531903    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr6tz\" (UniqueName: \"kubernetes.io/projected/6aad11a8-4cbe-45c2-8b98-cf11267f9300-kube-api-access-lr6tz\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5\" (UID: \"6aad11a8-4cbe-45c2-8b98-cf11267f9300\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"
Sep 16 17:03:01.532212 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.531976    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aad11a8-4cbe-45c2-8b98-cf11267f9300-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5\" (UID: \"6aad11a8-4cbe-45c2-8b98-cf11267f9300\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"
Sep 16 17:03:01.632798 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.632764    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr6tz\" (UniqueName: \"kubernetes.io/projected/6aad11a8-4cbe-45c2-8b98-cf11267f9300-kube-api-access-lr6tz\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5\" (UID: \"6aad11a8-4cbe-45c2-8b98-cf11267f9300\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"
Sep 16 17:03:01.632956 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.632812    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aad11a8-4cbe-45c2-8b98-cf11267f9300-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5\" (UID: \"6aad11a8-4cbe-45c2-8b98-cf11267f9300\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"
Sep 16 17:03:01.635216 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.635185    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aad11a8-4cbe-45c2-8b98-cf11267f9300-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5\" (UID: \"6aad11a8-4cbe-45c2-8b98-cf11267f9300\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"
Sep 16 17:03:01.642988 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.642964    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr6tz\" (UniqueName: \"kubernetes.io/projected/6aad11a8-4cbe-45c2-8b98-cf11267f9300-kube-api-access-lr6tz\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5\" (UID: \"6aad11a8-4cbe-45c2-8b98-cf11267f9300\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"
Sep 16 17:03:01.746769 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:01.746685    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"
Sep 16 17:03:02.020751 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:02.020671    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"]
Sep 16 17:03:02.024921 ip-10-0-143-143 kubenswrapper[2574]: W0916 17:03:02.024887    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aad11a8_4cbe_45c2_8b98_cf11267f9300.slice/crio-0a367a826516611f3a3627848546f98c789136d8eda7ca4b334ca503cd62de46 WatchSource:0}: Error finding container 0a367a826516611f3a3627848546f98c789136d8eda7ca4b334ca503cd62de46: Status 404 returned error can't find the container with id 0a367a826516611f3a3627848546f98c789136d8eda7ca4b334ca503cd62de46
Sep 16 17:03:02.578324 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:02.578281    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5" event={"ID":"6aad11a8-4cbe-45c2-8b98-cf11267f9300","Type":"ContainerStarted","Data":"eb7d226f6665ccd778c623b9badc9a0e7f145ee41c612df05b2f1450dda189d7"}
Sep 16 17:03:02.578756 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:02.578354    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5" event={"ID":"6aad11a8-4cbe-45c2-8b98-cf11267f9300","Type":"ContainerStarted","Data":"0a367a826516611f3a3627848546f98c789136d8eda7ca4b334ca503cd62de46"}
Sep 16 17:03:02.578756 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:02.578385    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"
Sep 16 17:03:02.595842 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:02.595789    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5" podStartSLOduration=1.595772894 podStartE2EDuration="1.595772894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:03:02.59336284 +0000 UTC m=+4379.112985616" watchObservedRunningTime="2026-09-16 17:03:02.595772894 +0000 UTC m=+4379.115395669"
Sep 16 17:03:07.236476 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:07.236433    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 17:03:17.236387 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:17.236338    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 17:03:27.235563 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:27.235365    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8001/health\": dial tcp 10.134.0.82:8001: connect: connection refused"
Sep 16 17:03:33.585402 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:33.585371    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"
Sep 16 17:03:37.252022 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:37.251980    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:03:37.270757 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:37.270727    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:03:46.240185 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:03:46.240146    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"]
Sep 16 17:04:01.262129 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:01.262085    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main" containerID="cri-o://515742277a6d636c3d618a84eabffb32485d7940754fd8188b29fd7cd76afe1f" gracePeriod=45
Sep 16 17:04:01.878176 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:01.878145    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="llm-d-routing-sidecar" containerID="cri-o://9c4be8b19eca3382fcc5d9ebef4c2edc8f234bd89d0e86e8b477e9d7f44f4cbd" gracePeriod=45
Sep 16 17:04:01.882574 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:01.882547    2574 generic.go:356] "Generic (PLEG): container finished" podID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerID="515742277a6d636c3d618a84eabffb32485d7940754fd8188b29fd7cd76afe1f" exitCode=0
Sep 16 17:04:01.882687 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:01.882595    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" event={"ID":"4a180882-99e0-4672-80ba-00b5d9753ce1","Type":"ContainerDied","Data":"515742277a6d636c3d618a84eabffb32485d7940754fd8188b29fd7cd76afe1f"}
Sep 16 17:04:02.117183 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.117158    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:04:02.224748 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.224702    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfqrp\" (UniqueName: \"kubernetes.io/projected/4a180882-99e0-4672-80ba-00b5d9753ce1-kube-api-access-tfqrp\") pod \"4a180882-99e0-4672-80ba-00b5d9753ce1\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") "
Sep 16 17:04:02.224951 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.224771    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-dshm\") pod \"4a180882-99e0-4672-80ba-00b5d9753ce1\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") "
Sep 16 17:04:02.224951 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.224805    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-tmp-dir\") pod \"4a180882-99e0-4672-80ba-00b5d9753ce1\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") "
Sep 16 17:04:02.224951 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.224847    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a180882-99e0-4672-80ba-00b5d9753ce1-tls-certs\") pod \"4a180882-99e0-4672-80ba-00b5d9753ce1\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") "
Sep 16 17:04:02.224951 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.224878    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-home\") pod \"4a180882-99e0-4672-80ba-00b5d9753ce1\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") "
Sep 16 17:04:02.224951 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.224902    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-kserve-provision-location\") pod \"4a180882-99e0-4672-80ba-00b5d9753ce1\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") "
Sep 16 17:04:02.225206 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.224958    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a180882-99e0-4672-80ba-00b5d9753ce1-cabundle-cert\") pod \"4a180882-99e0-4672-80ba-00b5d9753ce1\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") "
Sep 16 17:04:02.225206 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.225016    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-model-cache\") pod \"4a180882-99e0-4672-80ba-00b5d9753ce1\" (UID: \"4a180882-99e0-4672-80ba-00b5d9753ce1\") "
Sep 16 17:04:02.225578 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.225510    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a180882-99e0-4672-80ba-00b5d9753ce1" (UID: "4a180882-99e0-4672-80ba-00b5d9753ce1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:02.225578 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.225557    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-home" (OuterVolumeSpecName: "home") pod "4a180882-99e0-4672-80ba-00b5d9753ce1" (UID: "4a180882-99e0-4672-80ba-00b5d9753ce1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:02.226059 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.226039    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a180882-99e0-4672-80ba-00b5d9753ce1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a180882-99e0-4672-80ba-00b5d9753ce1" (UID: "4a180882-99e0-4672-80ba-00b5d9753ce1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:04:02.227130 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.227103    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-dshm" (OuterVolumeSpecName: "dshm") pod "4a180882-99e0-4672-80ba-00b5d9753ce1" (UID: "4a180882-99e0-4672-80ba-00b5d9753ce1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:02.227298 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.227280    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a180882-99e0-4672-80ba-00b5d9753ce1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a180882-99e0-4672-80ba-00b5d9753ce1" (UID: "4a180882-99e0-4672-80ba-00b5d9753ce1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:04:02.227383 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.227358    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a180882-99e0-4672-80ba-00b5d9753ce1-kube-api-access-tfqrp" (OuterVolumeSpecName: "kube-api-access-tfqrp") pod "4a180882-99e0-4672-80ba-00b5d9753ce1" (UID: "4a180882-99e0-4672-80ba-00b5d9753ce1"). InnerVolumeSpecName "kube-api-access-tfqrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:04:02.237459 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.237435    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a180882-99e0-4672-80ba-00b5d9753ce1" (UID: "4a180882-99e0-4672-80ba-00b5d9753ce1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:02.249598 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.249565    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a180882-99e0-4672-80ba-00b5d9753ce1" (UID: "4a180882-99e0-4672-80ba-00b5d9753ce1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:02.326209 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.326106    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a180882-99e0-4672-80ba-00b5d9753ce1-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.326209 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.326142    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.326209 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.326155    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfqrp\" (UniqueName: \"kubernetes.io/projected/4a180882-99e0-4672-80ba-00b5d9753ce1-kube-api-access-tfqrp\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.326209 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.326169    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.326209 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.326210    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.326784 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.326223    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a180882-99e0-4672-80ba-00b5d9753ce1-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.326784 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.326233    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.326784 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.326244    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a180882-99e0-4672-80ba-00b5d9753ce1-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:04:02.888991 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.888952    2574 generic.go:356] "Generic (PLEG): container finished" podID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerID="9c4be8b19eca3382fcc5d9ebef4c2edc8f234bd89d0e86e8b477e9d7f44f4cbd" exitCode=0
Sep 16 17:04:02.889192 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.889077    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"
Sep 16 17:04:02.889192 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.889083    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" event={"ID":"4a180882-99e0-4672-80ba-00b5d9753ce1","Type":"ContainerDied","Data":"9c4be8b19eca3382fcc5d9ebef4c2edc8f234bd89d0e86e8b477e9d7f44f4cbd"}
Sep 16 17:04:02.889192 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.889172    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz" event={"ID":"4a180882-99e0-4672-80ba-00b5d9753ce1","Type":"ContainerDied","Data":"5b8c157f340b7f74d529fb7499b6ca5b787f0df22907a9f942d066912c4da9e1"}
Sep 16 17:04:02.889372 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.889208    2574 scope.go:117] "RemoveContainer" containerID="515742277a6d636c3d618a84eabffb32485d7940754fd8188b29fd7cd76afe1f"
Sep 16 17:04:02.899677 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.899658    2574 scope.go:117] "RemoveContainer" containerID="7bd8ff93119f0a0b929b630025669da11be6d32a9420757c0749599085cddb72"
Sep 16 17:04:02.907639 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.907616    2574 scope.go:117] "RemoveContainer" containerID="9c4be8b19eca3382fcc5d9ebef4c2edc8f234bd89d0e86e8b477e9d7f44f4cbd"
Sep 16 17:04:02.913763 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.913738    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"]
Sep 16 17:04:02.917543 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.917504    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-9d6ff4d7f-h59vz"]
Sep 16 17:04:02.919559 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.919541    2574 scope.go:117] "RemoveContainer" containerID="515742277a6d636c3d618a84eabffb32485d7940754fd8188b29fd7cd76afe1f"
Sep 16 17:04:02.919875 ip-10-0-143-143 kubenswrapper[2574]: E0916 17:04:02.919858    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515742277a6d636c3d618a84eabffb32485d7940754fd8188b29fd7cd76afe1f\": container with ID starting with 515742277a6d636c3d618a84eabffb32485d7940754fd8188b29fd7cd76afe1f not found: ID does not exist" containerID="515742277a6d636c3d618a84eabffb32485d7940754fd8188b29fd7cd76afe1f"
Sep 16 17:04:02.919928 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.919884    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515742277a6d636c3d618a84eabffb32485d7940754fd8188b29fd7cd76afe1f"} err="failed to get container status \"515742277a6d636c3d618a84eabffb32485d7940754fd8188b29fd7cd76afe1f\": rpc error: code = NotFound desc = could not find container \"515742277a6d636c3d618a84eabffb32485d7940754fd8188b29fd7cd76afe1f\": container with ID starting with 515742277a6d636c3d618a84eabffb32485d7940754fd8188b29fd7cd76afe1f not found: ID does not exist"
Sep 16 17:04:02.919928 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.919901    2574 scope.go:117] "RemoveContainer" containerID="7bd8ff93119f0a0b929b630025669da11be6d32a9420757c0749599085cddb72"
Sep 16 17:04:02.920153 ip-10-0-143-143 kubenswrapper[2574]: E0916 17:04:02.920131    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd8ff93119f0a0b929b630025669da11be6d32a9420757c0749599085cddb72\": container with ID starting with 7bd8ff93119f0a0b929b630025669da11be6d32a9420757c0749599085cddb72 not found: ID does not exist" containerID="7bd8ff93119f0a0b929b630025669da11be6d32a9420757c0749599085cddb72"
Sep 16 17:04:02.920219 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.920163    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd8ff93119f0a0b929b630025669da11be6d32a9420757c0749599085cddb72"} err="failed to get container status \"7bd8ff93119f0a0b929b630025669da11be6d32a9420757c0749599085cddb72\": rpc error: code = NotFound desc = could not find container \"7bd8ff93119f0a0b929b630025669da11be6d32a9420757c0749599085cddb72\": container with ID starting with 7bd8ff93119f0a0b929b630025669da11be6d32a9420757c0749599085cddb72 not found: ID does not exist"
Sep 16 17:04:02.920219 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.920184    2574 scope.go:117] "RemoveContainer" containerID="9c4be8b19eca3382fcc5d9ebef4c2edc8f234bd89d0e86e8b477e9d7f44f4cbd"
Sep 16 17:04:02.920443 ip-10-0-143-143 kubenswrapper[2574]: E0916 17:04:02.920427    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4be8b19eca3382fcc5d9ebef4c2edc8f234bd89d0e86e8b477e9d7f44f4cbd\": container with ID starting with 9c4be8b19eca3382fcc5d9ebef4c2edc8f234bd89d0e86e8b477e9d7f44f4cbd not found: ID does not exist" containerID="9c4be8b19eca3382fcc5d9ebef4c2edc8f234bd89d0e86e8b477e9d7f44f4cbd"
Sep 16 17:04:02.920498 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:02.920449    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4be8b19eca3382fcc5d9ebef4c2edc8f234bd89d0e86e8b477e9d7f44f4cbd"} err="failed to get container status \"9c4be8b19eca3382fcc5d9ebef4c2edc8f234bd89d0e86e8b477e9d7f44f4cbd\": rpc error: code = NotFound desc = could not find container \"9c4be8b19eca3382fcc5d9ebef4c2edc8f234bd89d0e86e8b477e9d7f44f4cbd\": container with ID starting with 9c4be8b19eca3382fcc5d9ebef4c2edc8f234bd89d0e86e8b477e9d7f44f4cbd not found: ID does not exist"
Sep 16 17:04:03.995211 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:03.995174    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" path="/var/lib/kubelet/pods/4a180882-99e0-4672-80ba-00b5d9753ce1/volumes"
Sep 16 17:04:04.674425 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:04.674392    2574 scope.go:117] "RemoveContainer" containerID="19d966757b73753584634d59378df974e03c6cf7d8fca06b679563719aa053a9"
Sep 16 17:04:46.148410 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.148370    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"]
Sep 16 17:04:46.148876 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.148786    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="llm-d-routing-sidecar"
Sep 16 17:04:46.148876 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.148799    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="llm-d-routing-sidecar"
Sep 16 17:04:46.148876 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.148807    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="storage-initializer"
Sep 16 17:04:46.148876 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.148813    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="storage-initializer"
Sep 16 17:04:46.148876 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.148832    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main"
Sep 16 17:04:46.148876 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.148838    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main"
Sep 16 17:04:46.149068 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.148888    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="main"
Sep 16 17:04:46.149068 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.148897    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a180882-99e0-4672-80ba-00b5d9753ce1" containerName="llm-d-routing-sidecar"
Sep 16 17:04:46.152176 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.152157    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.154660 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.154622    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:04:46.154806 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.154666    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 17:04:46.154806 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.154678    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 17:04:46.154806 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.154686    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 17:04:46.154806 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.154743    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 17:04:46.154806 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.154746    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-qr8qg\""
Sep 16 17:04:46.166389 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.166360    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"]
Sep 16 17:04:46.236331 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.236297    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-home\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.236512 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.236337    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.236512 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.236359    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dafa10-efd3-40b7-a7a1-564c2a829b9e-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.236512 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.236402    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgkjg\" (UniqueName: \"kubernetes.io/projected/62dafa10-efd3-40b7-a7a1-564c2a829b9e-kube-api-access-cgkjg\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.236512 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.236426    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.236512 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.236447    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.236512 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.236474    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.236752 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.236539    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dafa10-efd3-40b7-a7a1-564c2a829b9e-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.337436 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.337395    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dafa10-efd3-40b7-a7a1-564c2a829b9e-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.337662 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.337470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-home\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.337662 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.337491    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.337662 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.337510    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dafa10-efd3-40b7-a7a1-564c2a829b9e-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.337662 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.337604    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgkjg\" (UniqueName: \"kubernetes.io/projected/62dafa10-efd3-40b7-a7a1-564c2a829b9e-kube-api-access-cgkjg\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.337662 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.337643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.337917 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.337672    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.337917 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.337715    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.338021 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.337946    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-home\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.338078 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.338016    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.338078 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.338037    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.338246 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.338225    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.338295 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.338271    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dafa10-efd3-40b7-a7a1-564c2a829b9e-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.339961 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.339938    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.340296 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.340279    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dafa10-efd3-40b7-a7a1-564c2a829b9e-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.345502 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.345475    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgkjg\" (UniqueName: \"kubernetes.io/projected/62dafa10-efd3-40b7-a7a1-564c2a829b9e-kube-api-access-cgkjg\") pod \"router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.462717 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.462617    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:46.741429 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:46.741401    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"]
Sep 16 17:04:46.743550 ip-10-0-143-143 kubenswrapper[2574]: W0916 17:04:46.743495    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62dafa10_efd3_40b7_a7a1_564c2a829b9e.slice/crio-a42d19e30f6b2597d6557ef255076baa51210cd40deecd09fbbdc8a24bcb5fe6 WatchSource:0}: Error finding container a42d19e30f6b2597d6557ef255076baa51210cd40deecd09fbbdc8a24bcb5fe6: Status 404 returned error can't find the container with id a42d19e30f6b2597d6557ef255076baa51210cd40deecd09fbbdc8a24bcb5fe6
Sep 16 17:04:47.102553 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:47.102504    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" event={"ID":"62dafa10-efd3-40b7-a7a1-564c2a829b9e","Type":"ContainerStarted","Data":"e191eb3c321cd9270767f0c31eaf3617c070b513276c1c4d388f4e52f252abb9"}
Sep 16 17:04:47.102832 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:47.102589    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" event={"ID":"62dafa10-efd3-40b7-a7a1-564c2a829b9e","Type":"ContainerStarted","Data":"a42d19e30f6b2597d6557ef255076baa51210cd40deecd09fbbdc8a24bcb5fe6"}
Sep 16 17:04:50.121024 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:50.120983    2574 generic.go:356] "Generic (PLEG): container finished" podID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerID="e191eb3c321cd9270767f0c31eaf3617c070b513276c1c4d388f4e52f252abb9" exitCode=0
Sep 16 17:04:50.121485 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:50.121067    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" event={"ID":"62dafa10-efd3-40b7-a7a1-564c2a829b9e","Type":"ContainerDied","Data":"e191eb3c321cd9270767f0c31eaf3617c070b513276c1c4d388f4e52f252abb9"}
Sep 16 17:04:51.127996 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:51.127962    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" event={"ID":"62dafa10-efd3-40b7-a7a1-564c2a829b9e","Type":"ContainerStarted","Data":"a915b0e9551d6d25044c92633b9a63bf70d70938694fc3e6a0f751f61d3066fe"}
Sep 16 17:04:51.150004 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:51.149950    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" podStartSLOduration=2.820119599 podStartE2EDuration="5.14993475s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.329815151s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:04:51.148012732 +0000 UTC m=+4487.667635509" watchObservedRunningTime="2026-09-16 17:04:51.14993475 +0000 UTC m=+4487.669557525"
Sep 16 17:04:56.463022 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:56.462920    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:56.463494 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:56.463024    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:04:56.464540 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:04:56.464472    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 16 17:05:06.209404 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:06.209366    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"]
Sep 16 17:05:06.463829 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:06.463729    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 16 17:05:13.585756 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:13.585718    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5" podUID="6aad11a8-4cbe-45c2-8b98-cf11267f9300" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:05:16.464023 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:16.463977    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 16 17:05:21.230206 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:21.230143    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5" podUID="6aad11a8-4cbe-45c2-8b98-cf11267f9300" containerName="main" containerID="cri-o://eb7d226f6665ccd778c623b9badc9a0e7f145ee41c612df05b2f1450dda189d7" gracePeriod=45
Sep 16 17:05:23.585643 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:23.585605    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5" podUID="6aad11a8-4cbe-45c2-8b98-cf11267f9300" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:05:26.463964 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:26.463910    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 16 17:05:33.585528 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:33.585475    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5" podUID="6aad11a8-4cbe-45c2-8b98-cf11267f9300" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:05:33.585958 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:33.585621    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"
Sep 16 17:05:36.463875 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:36.463828    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 16 17:05:43.585684 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:43.585647    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5" podUID="6aad11a8-4cbe-45c2-8b98-cf11267f9300" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:05:46.463477 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:46.463420    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 16 17:05:51.443884 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:51.443852    2574 generic.go:356] "Generic (PLEG): container finished" podID="6aad11a8-4cbe-45c2-8b98-cf11267f9300" containerID="eb7d226f6665ccd778c623b9badc9a0e7f145ee41c612df05b2f1450dda189d7" exitCode=0
Sep 16 17:05:51.444229 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:51.443905    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5" event={"ID":"6aad11a8-4cbe-45c2-8b98-cf11267f9300","Type":"ContainerDied","Data":"eb7d226f6665ccd778c623b9badc9a0e7f145ee41c612df05b2f1450dda189d7"}
Sep 16 17:05:51.491653 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:51.491623    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"
Sep 16 17:05:51.563711 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:51.563606    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr6tz\" (UniqueName: \"kubernetes.io/projected/6aad11a8-4cbe-45c2-8b98-cf11267f9300-kube-api-access-lr6tz\") pod \"6aad11a8-4cbe-45c2-8b98-cf11267f9300\" (UID: \"6aad11a8-4cbe-45c2-8b98-cf11267f9300\") "
Sep 16 17:05:51.563910 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:51.563884    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aad11a8-4cbe-45c2-8b98-cf11267f9300-tls-certs\") pod \"6aad11a8-4cbe-45c2-8b98-cf11267f9300\" (UID: \"6aad11a8-4cbe-45c2-8b98-cf11267f9300\") "
Sep 16 17:05:51.566059 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:51.566033    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aad11a8-4cbe-45c2-8b98-cf11267f9300-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6aad11a8-4cbe-45c2-8b98-cf11267f9300" (UID: "6aad11a8-4cbe-45c2-8b98-cf11267f9300"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:05:51.566182 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:51.566102    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aad11a8-4cbe-45c2-8b98-cf11267f9300-kube-api-access-lr6tz" (OuterVolumeSpecName: "kube-api-access-lr6tz") pod "6aad11a8-4cbe-45c2-8b98-cf11267f9300" (UID: "6aad11a8-4cbe-45c2-8b98-cf11267f9300"). InnerVolumeSpecName "kube-api-access-lr6tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:05:51.665233 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:51.665195    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr6tz\" (UniqueName: \"kubernetes.io/projected/6aad11a8-4cbe-45c2-8b98-cf11267f9300-kube-api-access-lr6tz\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:05:51.665233 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:51.665227    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aad11a8-4cbe-45c2-8b98-cf11267f9300-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:05:52.450950 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:52.450916    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"
Sep 16 17:05:52.451438 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:52.450946    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5" event={"ID":"6aad11a8-4cbe-45c2-8b98-cf11267f9300","Type":"ContainerDied","Data":"0a367a826516611f3a3627848546f98c789136d8eda7ca4b334ca503cd62de46"}
Sep 16 17:05:52.451438 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:52.451011    2574 scope.go:117] "RemoveContainer" containerID="eb7d226f6665ccd778c623b9badc9a0e7f145ee41c612df05b2f1450dda189d7"
Sep 16 17:05:52.466138 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:52.466113    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"]
Sep 16 17:05:52.471881 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:52.471853    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-sqfw5"]
Sep 16 17:05:53.994357 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:53.994322    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aad11a8-4cbe-45c2-8b98-cf11267f9300" path="/var/lib/kubelet/pods/6aad11a8-4cbe-45c2-8b98-cf11267f9300/volumes"
Sep 16 17:05:56.463849 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:05:56.463805    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 16 17:06:06.463757 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:06.463712    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 16 17:06:08.206636 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.206599    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"]
Sep 16 17:06:08.207235 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.207212    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aad11a8-4cbe-45c2-8b98-cf11267f9300" containerName="main"
Sep 16 17:06:08.207344 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.207238    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aad11a8-4cbe-45c2-8b98-cf11267f9300" containerName="main"
Sep 16 17:06:08.207401 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.207342    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aad11a8-4cbe-45c2-8b98-cf11267f9300" containerName="main"
Sep 16 17:06:08.212022 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.211996    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"
Sep 16 17:06:08.216026 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.216000    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 17:06:08.216238 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.216006    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-zmd74\""
Sep 16 17:06:08.216352 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.216014    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 17:06:08.216352 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.216022    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-tznph\""
Sep 16 17:06:08.216352 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.216039    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 17:06:08.223331 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.223306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6pks\" (UniqueName: \"kubernetes.io/projected/8b3cfd4a-0381-451a-a65a-085318c6d5e1-kube-api-access-l6pks\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57\" (UID: \"8b3cfd4a-0381-451a-a65a-085318c6d5e1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"
Sep 16 17:06:08.223509 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.223485    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3cfd4a-0381-451a-a65a-085318c6d5e1-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57\" (UID: \"8b3cfd4a-0381-451a-a65a-085318c6d5e1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"
Sep 16 17:06:08.224562 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.224539    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"]
Sep 16 17:06:08.325077 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.325031    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3cfd4a-0381-451a-a65a-085318c6d5e1-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57\" (UID: \"8b3cfd4a-0381-451a-a65a-085318c6d5e1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"
Sep 16 17:06:08.325270 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.325090    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6pks\" (UniqueName: \"kubernetes.io/projected/8b3cfd4a-0381-451a-a65a-085318c6d5e1-kube-api-access-l6pks\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57\" (UID: \"8b3cfd4a-0381-451a-a65a-085318c6d5e1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"
Sep 16 17:06:08.327693 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.327670    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3cfd4a-0381-451a-a65a-085318c6d5e1-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57\" (UID: \"8b3cfd4a-0381-451a-a65a-085318c6d5e1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"
Sep 16 17:06:08.334996 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.334967    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6pks\" (UniqueName: \"kubernetes.io/projected/8b3cfd4a-0381-451a-a65a-085318c6d5e1-kube-api-access-l6pks\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57\" (UID: \"8b3cfd4a-0381-451a-a65a-085318c6d5e1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"
Sep 16 17:06:08.523229 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.523131    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"
Sep 16 17:06:08.799884 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:08.799803    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"]
Sep 16 17:06:08.803338 ip-10-0-143-143 kubenswrapper[2574]: W0916 17:06:08.803304    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3cfd4a_0381_451a_a65a_085318c6d5e1.slice/crio-20d994a60168090476b8c3a209864382d2d4348a9cba34f28fb02206642983d2 WatchSource:0}: Error finding container 20d994a60168090476b8c3a209864382d2d4348a9cba34f28fb02206642983d2: Status 404 returned error can't find the container with id 20d994a60168090476b8c3a209864382d2d4348a9cba34f28fb02206642983d2
Sep 16 17:06:09.548954 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:09.548914    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57" event={"ID":"8b3cfd4a-0381-451a-a65a-085318c6d5e1","Type":"ContainerStarted","Data":"59d276849b9adc3913cff742b4223548f1e7c071672748ad8d042850afdd4f0f"}
Sep 16 17:06:09.549349 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:09.548968    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57" event={"ID":"8b3cfd4a-0381-451a-a65a-085318c6d5e1","Type":"ContainerStarted","Data":"20d994a60168090476b8c3a209864382d2d4348a9cba34f28fb02206642983d2"}
Sep 16 17:06:09.549349 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:09.548998    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"
Sep 16 17:06:09.567658 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:09.567611    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57" podStartSLOduration=1.567595994 podStartE2EDuration="1.567595994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:06:09.565317894 +0000 UTC m=+4566.084940679" watchObservedRunningTime="2026-09-16 17:06:09.567595994 +0000 UTC m=+4566.087218770"
Sep 16 17:06:16.464011 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:16.463972    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 16 17:06:26.463683 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:26.463595    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 16 17:06:36.463617 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:36.463563    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 16 17:06:40.556752 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:40.556720    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"
Sep 16 17:06:46.473785 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:46.473751    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:06:46.481378 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:46.481348    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:06:55.750528 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:06:55.750479    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"]
Sep 16 17:07:04.720055 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:04.720022    2574 scope.go:117] "RemoveContainer" containerID="bc260c6e77325bbc04ea8a5c44e0835b8fa6c735d13f2ed3be5212efe79488b9"
Sep 16 17:07:10.772857 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:10.772818    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main" containerID="cri-o://a915b0e9551d6d25044c92633b9a63bf70d70938694fc3e6a0f751f61d3066fe" gracePeriod=45
Sep 16 17:07:11.521477 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.521447    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:07:11.611084 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.611060    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-dshm\") pod \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") "
Sep 16 17:07:11.611247 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.611099    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-tmp-dir\") pod \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") "
Sep 16 17:07:11.611247 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.611130    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dafa10-efd3-40b7-a7a1-564c2a829b9e-cabundle-cert\") pod \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") "
Sep 16 17:07:11.611247 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.611160    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-model-cache\") pod \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") "
Sep 16 17:07:11.611247 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.611211    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-home\") pod \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") "
Sep 16 17:07:11.611473 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.611257    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-kserve-provision-location\") pod \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") "
Sep 16 17:07:11.611473 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.611283    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dafa10-efd3-40b7-a7a1-564c2a829b9e-tls-certs\") pod \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") "
Sep 16 17:07:11.611473 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.611383    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgkjg\" (UniqueName: \"kubernetes.io/projected/62dafa10-efd3-40b7-a7a1-564c2a829b9e-kube-api-access-cgkjg\") pod \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\" (UID: \"62dafa10-efd3-40b7-a7a1-564c2a829b9e\") "
Sep 16 17:07:11.611668 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.611508    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62dafa10-efd3-40b7-a7a1-564c2a829b9e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62dafa10-efd3-40b7-a7a1-564c2a829b9e" (UID: "62dafa10-efd3-40b7-a7a1-564c2a829b9e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:07:11.611833 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.611814    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dafa10-efd3-40b7-a7a1-564c2a829b9e-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.612052 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.612025    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-home" (OuterVolumeSpecName: "home") pod "62dafa10-efd3-40b7-a7a1-564c2a829b9e" (UID: "62dafa10-efd3-40b7-a7a1-564c2a829b9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:11.612155 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.612134    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "62dafa10-efd3-40b7-a7a1-564c2a829b9e" (UID: "62dafa10-efd3-40b7-a7a1-564c2a829b9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:11.613618 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.613590    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-dshm" (OuterVolumeSpecName: "dshm") pod "62dafa10-efd3-40b7-a7a1-564c2a829b9e" (UID: "62dafa10-efd3-40b7-a7a1-564c2a829b9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:11.614042 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.614012    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dafa10-efd3-40b7-a7a1-564c2a829b9e-kube-api-access-cgkjg" (OuterVolumeSpecName: "kube-api-access-cgkjg") pod "62dafa10-efd3-40b7-a7a1-564c2a829b9e" (UID: "62dafa10-efd3-40b7-a7a1-564c2a829b9e"). InnerVolumeSpecName "kube-api-access-cgkjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:07:11.614131 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.614058    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62dafa10-efd3-40b7-a7a1-564c2a829b9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62dafa10-efd3-40b7-a7a1-564c2a829b9e" (UID: "62dafa10-efd3-40b7-a7a1-564c2a829b9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:07:11.629624 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.629596    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62dafa10-efd3-40b7-a7a1-564c2a829b9e" (UID: "62dafa10-efd3-40b7-a7a1-564c2a829b9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:11.643548 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.643507    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62dafa10-efd3-40b7-a7a1-564c2a829b9e" (UID: "62dafa10-efd3-40b7-a7a1-564c2a829b9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:11.712402 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.712376    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.712402 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.712400    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dafa10-efd3-40b7-a7a1-564c2a829b9e-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.712556 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.712410    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgkjg\" (UniqueName: \"kubernetes.io/projected/62dafa10-efd3-40b7-a7a1-564c2a829b9e-kube-api-access-cgkjg\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.712556 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.712419    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.712556 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.712429    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.712556 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.712437    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.712556 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.712444    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dafa10-efd3-40b7-a7a1-564c2a829b9e-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:07:11.856142 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.856112    2574 generic.go:356] "Generic (PLEG): container finished" podID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerID="a915b0e9551d6d25044c92633b9a63bf70d70938694fc3e6a0f751f61d3066fe" exitCode=0
Sep 16 17:07:11.856510 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.856164    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"
Sep 16 17:07:11.856510 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.856202    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" event={"ID":"62dafa10-efd3-40b7-a7a1-564c2a829b9e","Type":"ContainerDied","Data":"a915b0e9551d6d25044c92633b9a63bf70d70938694fc3e6a0f751f61d3066fe"}
Sep 16 17:07:11.856510 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.856260    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d" event={"ID":"62dafa10-efd3-40b7-a7a1-564c2a829b9e","Type":"ContainerDied","Data":"a42d19e30f6b2597d6557ef255076baa51210cd40deecd09fbbdc8a24bcb5fe6"}
Sep 16 17:07:11.856510 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.856282    2574 scope.go:117] "RemoveContainer" containerID="a915b0e9551d6d25044c92633b9a63bf70d70938694fc3e6a0f751f61d3066fe"
Sep 16 17:07:11.865239 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.865223    2574 scope.go:117] "RemoveContainer" containerID="e191eb3c321cd9270767f0c31eaf3617c070b513276c1c4d388f4e52f252abb9"
Sep 16 17:07:11.872509 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.872491    2574 scope.go:117] "RemoveContainer" containerID="a915b0e9551d6d25044c92633b9a63bf70d70938694fc3e6a0f751f61d3066fe"
Sep 16 17:07:11.872783 ip-10-0-143-143 kubenswrapper[2574]: E0916 17:07:11.872766    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a915b0e9551d6d25044c92633b9a63bf70d70938694fc3e6a0f751f61d3066fe\": container with ID starting with a915b0e9551d6d25044c92633b9a63bf70d70938694fc3e6a0f751f61d3066fe not found: ID does not exist" containerID="a915b0e9551d6d25044c92633b9a63bf70d70938694fc3e6a0f751f61d3066fe"
Sep 16 17:07:11.872821 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.872790    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a915b0e9551d6d25044c92633b9a63bf70d70938694fc3e6a0f751f61d3066fe"} err="failed to get container status \"a915b0e9551d6d25044c92633b9a63bf70d70938694fc3e6a0f751f61d3066fe\": rpc error: code = NotFound desc = could not find container \"a915b0e9551d6d25044c92633b9a63bf70d70938694fc3e6a0f751f61d3066fe\": container with ID starting with a915b0e9551d6d25044c92633b9a63bf70d70938694fc3e6a0f751f61d3066fe not found: ID does not exist"
Sep 16 17:07:11.872821 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.872807    2574 scope.go:117] "RemoveContainer" containerID="e191eb3c321cd9270767f0c31eaf3617c070b513276c1c4d388f4e52f252abb9"
Sep 16 17:07:11.873039 ip-10-0-143-143 kubenswrapper[2574]: E0916 17:07:11.873022    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e191eb3c321cd9270767f0c31eaf3617c070b513276c1c4d388f4e52f252abb9\": container with ID starting with e191eb3c321cd9270767f0c31eaf3617c070b513276c1c4d388f4e52f252abb9 not found: ID does not exist" containerID="e191eb3c321cd9270767f0c31eaf3617c070b513276c1c4d388f4e52f252abb9"
Sep 16 17:07:11.873075 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.873046    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e191eb3c321cd9270767f0c31eaf3617c070b513276c1c4d388f4e52f252abb9"} err="failed to get container status \"e191eb3c321cd9270767f0c31eaf3617c070b513276c1c4d388f4e52f252abb9\": rpc error: code = NotFound desc = could not find container \"e191eb3c321cd9270767f0c31eaf3617c070b513276c1c4d388f4e52f252abb9\": container with ID starting with e191eb3c321cd9270767f0c31eaf3617c070b513276c1c4d388f4e52f252abb9 not found: ID does not exist"
Sep 16 17:07:11.877931 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.877911    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"]
Sep 16 17:07:11.882393 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.882371    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-85b68bd899-9748d"]
Sep 16 17:07:11.994275 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:11.994242    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" path="/var/lib/kubelet/pods/62dafa10-efd3-40b7-a7a1-564c2a829b9e/volumes"
Sep 16 17:07:51.398829 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.398794    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"]
Sep 16 17:07:51.399239 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.399186    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="storage-initializer"
Sep 16 17:07:51.399239 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.399199    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="storage-initializer"
Sep 16 17:07:51.399239 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.399213    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main"
Sep 16 17:07:51.399239 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.399221    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main"
Sep 16 17:07:51.399369 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.399296    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="62dafa10-efd3-40b7-a7a1-564c2a829b9e" containerName="main"
Sep 16 17:07:51.402701 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.402683    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.409603 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.409568    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:07:51.409803 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.409614    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 16 17:07:51.409946 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.409836    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-nt2h8\""
Sep 16 17:07:51.409946 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.409701    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 16 17:07:51.409946 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.409721    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 16 17:07:51.410441 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.410422    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 16 17:07:51.413540 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.413502    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"]
Sep 16 17:07:51.445412 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.445378    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.445608 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.445440    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.445608 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.445543    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.445608 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.445587    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.445755 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.445626    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wwkm\" (UniqueName: \"kubernetes.io/projected/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-kube-api-access-4wwkm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.445755 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.445651    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.445827 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.445793    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.445863 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.445849    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.546950 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.546913    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.547128 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.546963    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.547128 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.547007    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.547128 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.547043    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.547128 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.547071    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.547128 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.547107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wwkm\" (UniqueName: \"kubernetes.io/projected/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-kube-api-access-4wwkm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.547394 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.547131    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.547394 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.547211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.547394 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.547347    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.547600 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.547386    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.547600 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.547450    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.547688 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.547599    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.547810 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.547790    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.549442 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.549415    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.549617 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.549601    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.554838 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.554818    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wwkm\" (UniqueName: \"kubernetes.io/projected/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-kube-api-access-4wwkm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:51.721582 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:51.721473    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:07:52.014139 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:52.014110    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"]
Sep 16 17:07:52.015333 ip-10-0-143-143 kubenswrapper[2574]: W0916 17:07:52.015297    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a4e97c_9ba1_4e84_b577_479f3eb333e3.slice/crio-645441e90e2b26b50a5c50be36502737b6db8cfbe72dbaf80da3754e3be2fda2 WatchSource:0}: Error finding container 645441e90e2b26b50a5c50be36502737b6db8cfbe72dbaf80da3754e3be2fda2: Status 404 returned error can't find the container with id 645441e90e2b26b50a5c50be36502737b6db8cfbe72dbaf80da3754e3be2fda2
Sep 16 17:07:52.054501 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:52.054460    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" event={"ID":"d5a4e97c-9ba1-4e84-b577-479f3eb333e3","Type":"ContainerStarted","Data":"645441e90e2b26b50a5c50be36502737b6db8cfbe72dbaf80da3754e3be2fda2"}
Sep 16 17:07:53.061548 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:53.061491    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" event={"ID":"d5a4e97c-9ba1-4e84-b577-479f3eb333e3","Type":"ContainerStarted","Data":"ff728d10bcc95d531b8e61db318bcfc95ec7ec35abc4eb0909075388b4bcc34a"}
Sep 16 17:07:55.076116 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:55.076081    2574 generic.go:356] "Generic (PLEG): container finished" podID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerID="ff728d10bcc95d531b8e61db318bcfc95ec7ec35abc4eb0909075388b4bcc34a" exitCode=0
Sep 16 17:07:55.076494 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:55.076152    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" event={"ID":"d5a4e97c-9ba1-4e84-b577-479f3eb333e3","Type":"ContainerDied","Data":"ff728d10bcc95d531b8e61db318bcfc95ec7ec35abc4eb0909075388b4bcc34a"}
Sep 16 17:07:56.083456 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:56.083377    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" event={"ID":"d5a4e97c-9ba1-4e84-b577-479f3eb333e3","Type":"ContainerStarted","Data":"9b77c85f3be347da59f78351dd1c4217c4c2abd8f0a0cde1451ebeb64a09e294"}
Sep 16 17:07:56.106580 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:56.106502    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" podStartSLOduration=2.7889569119999997 podStartE2EDuration="5.106485719s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.317528807s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:07:56.104045923 +0000 UTC m=+4672.623668698" watchObservedRunningTime="2026-09-16 17:07:56.106485719 +0000 UTC m=+4672.626108494"
Sep 16 17:07:56.721703 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:07:56.721665    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:08:01.722350 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:08:01.722311    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:08:01.723705 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:08:01.723673    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": dial tcp 10.134.0.86:8000: connect: connection refused"
Sep 16 17:08:11.722843 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:08:11.722799    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": dial tcp 10.134.0.86:8000: connect: connection refused"
Sep 16 17:08:21.722307 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:08:21.722263    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": dial tcp 10.134.0.86:8000: connect: connection refused"
Sep 16 17:08:28.636043 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:08:28.636004    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"]
Sep 16 17:08:30.557160 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:08:30.557118    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57" podUID="8b3cfd4a-0381-451a-a65a-085318c6d5e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:08:31.722275 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:08:31.722237    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": dial tcp 10.134.0.86:8000: connect: connection refused"
Sep 16 17:08:40.556476 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:08:40.556425    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57" podUID="8b3cfd4a-0381-451a-a65a-085318c6d5e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:08:41.722103 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:08:41.722053    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": dial tcp 10.134.0.86:8000: connect: connection refused"
Sep 16 17:08:43.654931 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:08:43.654888    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57" podUID="8b3cfd4a-0381-451a-a65a-085318c6d5e1" containerName="main" containerID="cri-o://59d276849b9adc3913cff742b4223548f1e7c071672748ad8d042850afdd4f0f" gracePeriod=45
Sep 16 17:08:50.556786 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:08:50.556748    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57" podUID="8b3cfd4a-0381-451a-a65a-085318c6d5e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:08:50.557166 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:08:50.556859    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"
Sep 16 17:08:51.722612 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:08:51.722563    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": dial tcp 10.134.0.86:8000: connect: connection refused"
Sep 16 17:09:00.556879 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:00.556838    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57" podUID="8b3cfd4a-0381-451a-a65a-085318c6d5e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:09:01.722714 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:01.722668    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": dial tcp 10.134.0.86:8000: connect: connection refused"
Sep 16 17:09:10.556454 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:10.556411    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57" podUID="8b3cfd4a-0381-451a-a65a-085318c6d5e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:09:11.722482 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:11.722435    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": dial tcp 10.134.0.86:8000: connect: connection refused"
Sep 16 17:09:13.910040 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:13.910005    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"
Sep 16 17:09:13.945163 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:13.945087    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6pks\" (UniqueName: \"kubernetes.io/projected/8b3cfd4a-0381-451a-a65a-085318c6d5e1-kube-api-access-l6pks\") pod \"8b3cfd4a-0381-451a-a65a-085318c6d5e1\" (UID: \"8b3cfd4a-0381-451a-a65a-085318c6d5e1\") "
Sep 16 17:09:13.945163 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:13.945144    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3cfd4a-0381-451a-a65a-085318c6d5e1-tls-certs\") pod \"8b3cfd4a-0381-451a-a65a-085318c6d5e1\" (UID: \"8b3cfd4a-0381-451a-a65a-085318c6d5e1\") "
Sep 16 17:09:13.947349 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:13.947317    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b3cfd4a-0381-451a-a65a-085318c6d5e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b3cfd4a-0381-451a-a65a-085318c6d5e1" (UID: "8b3cfd4a-0381-451a-a65a-085318c6d5e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:09:13.947451 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:13.947360    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3cfd4a-0381-451a-a65a-085318c6d5e1-kube-api-access-l6pks" (OuterVolumeSpecName: "kube-api-access-l6pks") pod "8b3cfd4a-0381-451a-a65a-085318c6d5e1" (UID: "8b3cfd4a-0381-451a-a65a-085318c6d5e1"). InnerVolumeSpecName "kube-api-access-l6pks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:09:14.046602 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:14.046571    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6pks\" (UniqueName: \"kubernetes.io/projected/8b3cfd4a-0381-451a-a65a-085318c6d5e1-kube-api-access-l6pks\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:09:14.046602 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:14.046602    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3cfd4a-0381-451a-a65a-085318c6d5e1-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:09:14.490248 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:14.490215    2574 generic.go:356] "Generic (PLEG): container finished" podID="8b3cfd4a-0381-451a-a65a-085318c6d5e1" containerID="59d276849b9adc3913cff742b4223548f1e7c071672748ad8d042850afdd4f0f" exitCode=0
Sep 16 17:09:14.490412 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:14.490289    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57" event={"ID":"8b3cfd4a-0381-451a-a65a-085318c6d5e1","Type":"ContainerDied","Data":"59d276849b9adc3913cff742b4223548f1e7c071672748ad8d042850afdd4f0f"}
Sep 16 17:09:14.490412 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:14.490313    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"
Sep 16 17:09:14.490412 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:14.490339    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57" event={"ID":"8b3cfd4a-0381-451a-a65a-085318c6d5e1","Type":"ContainerDied","Data":"20d994a60168090476b8c3a209864382d2d4348a9cba34f28fb02206642983d2"}
Sep 16 17:09:14.490412 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:14.490368    2574 scope.go:117] "RemoveContainer" containerID="59d276849b9adc3913cff742b4223548f1e7c071672748ad8d042850afdd4f0f"
Sep 16 17:09:14.499191 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:14.499161    2574 scope.go:117] "RemoveContainer" containerID="59d276849b9adc3913cff742b4223548f1e7c071672748ad8d042850afdd4f0f"
Sep 16 17:09:14.499458 ip-10-0-143-143 kubenswrapper[2574]: E0916 17:09:14.499440    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d276849b9adc3913cff742b4223548f1e7c071672748ad8d042850afdd4f0f\": container with ID starting with 59d276849b9adc3913cff742b4223548f1e7c071672748ad8d042850afdd4f0f not found: ID does not exist" containerID="59d276849b9adc3913cff742b4223548f1e7c071672748ad8d042850afdd4f0f"
Sep 16 17:09:14.499556 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:14.499464    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d276849b9adc3913cff742b4223548f1e7c071672748ad8d042850afdd4f0f"} err="failed to get container status \"59d276849b9adc3913cff742b4223548f1e7c071672748ad8d042850afdd4f0f\": rpc error: code = NotFound desc = could not find container \"59d276849b9adc3913cff742b4223548f1e7c071672748ad8d042850afdd4f0f\": container with ID starting with 59d276849b9adc3913cff742b4223548f1e7c071672748ad8d042850afdd4f0f not found: ID does not exist"
Sep 16 17:09:14.506934 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:14.506907    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"]
Sep 16 17:09:14.511462 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:14.511441    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-rrb57"]
Sep 16 17:09:15.995926 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:15.995894    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b3cfd4a-0381-451a-a65a-085318c6d5e1" path="/var/lib/kubelet/pods/8b3cfd4a-0381-451a-a65a-085318c6d5e1/volumes"
Sep 16 17:09:21.722987 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:21.722943    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": dial tcp 10.134.0.86:8000: connect: connection refused"
Sep 16 17:09:31.722826 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:31.722739    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": dial tcp 10.134.0.86:8000: connect: connection refused"
Sep 16 17:09:41.722909 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:41.722859    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": dial tcp 10.134.0.86:8000: connect: connection refused"
Sep 16 17:09:51.732456 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:51.732421    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:09:51.740289 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:51.740262    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:09:58.610361 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:09:58.610319    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"]
Sep 16 17:10:13.631815 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:13.631774    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main" containerID="cri-o://9b77c85f3be347da59f78351dd1c4217c4c2abd8f0a0cde1451ebeb64a09e294" gracePeriod=45
Sep 16 17:10:13.732977 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:13.732939    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": dial tcp 10.134.0.86:8000: connect: connection refused"
Sep 16 17:10:14.374813 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.374789    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:10:14.493834 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.493752    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-home\") pod \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") "
Sep 16 17:10:14.493834 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.493813    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-dshm\") pod \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") "
Sep 16 17:10:14.494058 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.493847    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-kserve-provision-location\") pod \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") "
Sep 16 17:10:14.494058 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.493878    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-tmp-dir\") pod \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") "
Sep 16 17:10:14.494058 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.493908    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-cabundle-cert\") pod \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") "
Sep 16 17:10:14.494058 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.493958    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-model-cache\") pod \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") "
Sep 16 17:10:14.494058 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.493991    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-tls-certs\") pod \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") "
Sep 16 17:10:14.494058 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.494024    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wwkm\" (UniqueName: \"kubernetes.io/projected/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-kube-api-access-4wwkm\") pod \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\" (UID: \"d5a4e97c-9ba1-4e84-b577-479f3eb333e3\") "
Sep 16 17:10:14.494380 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.494259    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5a4e97c-9ba1-4e84-b577-479f3eb333e3" (UID: "d5a4e97c-9ba1-4e84-b577-479f3eb333e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:10:14.494455 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.494329    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5a4e97c-9ba1-4e84-b577-479f3eb333e3" (UID: "d5a4e97c-9ba1-4e84-b577-479f3eb333e3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:10:14.494508 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.494472    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-home" (OuterVolumeSpecName: "home") pod "d5a4e97c-9ba1-4e84-b577-479f3eb333e3" (UID: "d5a4e97c-9ba1-4e84-b577-479f3eb333e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:10:14.496142 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.496117    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5a4e97c-9ba1-4e84-b577-479f3eb333e3" (UID: "d5a4e97c-9ba1-4e84-b577-479f3eb333e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:10:14.496295 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.496277    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-kube-api-access-4wwkm" (OuterVolumeSpecName: "kube-api-access-4wwkm") pod "d5a4e97c-9ba1-4e84-b577-479f3eb333e3" (UID: "d5a4e97c-9ba1-4e84-b577-479f3eb333e3"). InnerVolumeSpecName "kube-api-access-4wwkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:10:14.496351 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.496319    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-dshm" (OuterVolumeSpecName: "dshm") pod "d5a4e97c-9ba1-4e84-b577-479f3eb333e3" (UID: "d5a4e97c-9ba1-4e84-b577-479f3eb333e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:10:14.506175 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.506151    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5a4e97c-9ba1-4e84-b577-479f3eb333e3" (UID: "d5a4e97c-9ba1-4e84-b577-479f3eb333e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:10:14.518398 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.518370    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5a4e97c-9ba1-4e84-b577-479f3eb333e3" (UID: "d5a4e97c-9ba1-4e84-b577-479f3eb333e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:10:14.594915 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.594892    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:10:14.594915 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.594915    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wwkm\" (UniqueName: \"kubernetes.io/projected/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-kube-api-access-4wwkm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:10:14.595041 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.594926    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:10:14.595041 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.594935    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:10:14.595041 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.594943    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-kserve-provision-location\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:10:14.595041 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.594967    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:10:14.595041 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.594977    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-cabundle-cert\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:10:14.595041 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.594986    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5a4e97c-9ba1-4e84-b577-479f3eb333e3-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:10:14.789708 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.789622    2574 generic.go:356] "Generic (PLEG): container finished" podID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerID="9b77c85f3be347da59f78351dd1c4217c4c2abd8f0a0cde1451ebeb64a09e294" exitCode=0
Sep 16 17:10:14.789708 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.789680    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"
Sep 16 17:10:14.790082 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.789709    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" event={"ID":"d5a4e97c-9ba1-4e84-b577-479f3eb333e3","Type":"ContainerDied","Data":"9b77c85f3be347da59f78351dd1c4217c4c2abd8f0a0cde1451ebeb64a09e294"}
Sep 16 17:10:14.790082 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.789766    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb" event={"ID":"d5a4e97c-9ba1-4e84-b577-479f3eb333e3","Type":"ContainerDied","Data":"645441e90e2b26b50a5c50be36502737b6db8cfbe72dbaf80da3754e3be2fda2"}
Sep 16 17:10:14.790082 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.789793    2574 scope.go:117] "RemoveContainer" containerID="9b77c85f3be347da59f78351dd1c4217c4c2abd8f0a0cde1451ebeb64a09e294"
Sep 16 17:10:14.798665 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.798647    2574 scope.go:117] "RemoveContainer" containerID="ff728d10bcc95d531b8e61db318bcfc95ec7ec35abc4eb0909075388b4bcc34a"
Sep 16 17:10:14.808236 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.808209    2574 scope.go:117] "RemoveContainer" containerID="9b77c85f3be347da59f78351dd1c4217c4c2abd8f0a0cde1451ebeb64a09e294"
Sep 16 17:10:14.808547 ip-10-0-143-143 kubenswrapper[2574]: E0916 17:10:14.808507    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b77c85f3be347da59f78351dd1c4217c4c2abd8f0a0cde1451ebeb64a09e294\": container with ID starting with 9b77c85f3be347da59f78351dd1c4217c4c2abd8f0a0cde1451ebeb64a09e294 not found: ID does not exist" containerID="9b77c85f3be347da59f78351dd1c4217c4c2abd8f0a0cde1451ebeb64a09e294"
Sep 16 17:10:14.808643 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.808558    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b77c85f3be347da59f78351dd1c4217c4c2abd8f0a0cde1451ebeb64a09e294"} err="failed to get container status \"9b77c85f3be347da59f78351dd1c4217c4c2abd8f0a0cde1451ebeb64a09e294\": rpc error: code = NotFound desc = could not find container \"9b77c85f3be347da59f78351dd1c4217c4c2abd8f0a0cde1451ebeb64a09e294\": container with ID starting with 9b77c85f3be347da59f78351dd1c4217c4c2abd8f0a0cde1451ebeb64a09e294 not found: ID does not exist"
Sep 16 17:10:14.808643 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.808575    2574 scope.go:117] "RemoveContainer" containerID="ff728d10bcc95d531b8e61db318bcfc95ec7ec35abc4eb0909075388b4bcc34a"
Sep 16 17:10:14.808836 ip-10-0-143-143 kubenswrapper[2574]: E0916 17:10:14.808820    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff728d10bcc95d531b8e61db318bcfc95ec7ec35abc4eb0909075388b4bcc34a\": container with ID starting with ff728d10bcc95d531b8e61db318bcfc95ec7ec35abc4eb0909075388b4bcc34a not found: ID does not exist" containerID="ff728d10bcc95d531b8e61db318bcfc95ec7ec35abc4eb0909075388b4bcc34a"
Sep 16 17:10:14.808896 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.808841    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff728d10bcc95d531b8e61db318bcfc95ec7ec35abc4eb0909075388b4bcc34a"} err="failed to get container status \"ff728d10bcc95d531b8e61db318bcfc95ec7ec35abc4eb0909075388b4bcc34a\": rpc error: code = NotFound desc = could not find container \"ff728d10bcc95d531b8e61db318bcfc95ec7ec35abc4eb0909075388b4bcc34a\": container with ID starting with ff728d10bcc95d531b8e61db318bcfc95ec7ec35abc4eb0909075388b4bcc34a not found: ID does not exist"
Sep 16 17:10:14.811264 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.811241    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"]
Sep 16 17:10:14.816697 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:14.816676    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c5b778dfdlnhtb"]
Sep 16 17:10:15.993701 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:15.993668    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" path="/var/lib/kubelet/pods/d5a4e97c-9ba1-4e84-b577-479f3eb333e3/volumes"
Sep 16 17:10:17.995653 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:17.995623    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"]
Sep 16 17:10:17.996092 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:17.995978    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="storage-initializer"
Sep 16 17:10:17.996092 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:17.995990    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="storage-initializer"
Sep 16 17:10:17.996092 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:17.995999    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b3cfd4a-0381-451a-a65a-085318c6d5e1" containerName="main"
Sep 16 17:10:17.996092 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:17.996006    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b3cfd4a-0381-451a-a65a-085318c6d5e1" containerName="main"
Sep 16 17:10:17.996092 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:17.996018    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main"
Sep 16 17:10:17.996092 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:17.996023    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main"
Sep 16 17:10:17.996092 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:17.996092    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b3cfd4a-0381-451a-a65a-085318c6d5e1" containerName="main"
Sep 16 17:10:17.996350 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:17.996101    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a4e97c-9ba1-4e84-b577-479f3eb333e3" containerName="main"
Sep 16 17:10:18.007511 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.007485    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.009912 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.009884    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 17:10:18.010035 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.009927    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"]
Sep 16 17:10:18.010553 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.010532    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-jbn9c\""
Sep 16 17:10:18.010553 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.010545    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 17:10:18.010717 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.010574    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 17:10:18.125370 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.125344    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.125495 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.125412    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92133754-268b-4699-b65b-23af1d99fbd1-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.125495 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.125444    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.125495 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.125465    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9qt\" (UniqueName: \"kubernetes.io/projected/92133754-268b-4699-b65b-23af1d99fbd1-kube-api-access-4t9qt\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.125649 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.125606    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.125649 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.125634    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.226638 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.226611    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.226770 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.226643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.226770 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.226669    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.226770 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.226702    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92133754-268b-4699-b65b-23af1d99fbd1-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.226770 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.226736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.226770 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.226758    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9qt\" (UniqueName: \"kubernetes.io/projected/92133754-268b-4699-b65b-23af1d99fbd1-kube-api-access-4t9qt\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.227080 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.227057    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.227136 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.227077    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.227176 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.227149    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.229457 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.229436    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.229875 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.229855    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92133754-268b-4699-b65b-23af1d99fbd1-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.234138 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.234114    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9qt\" (UniqueName: \"kubernetes.io/projected/92133754-268b-4699-b65b-23af1d99fbd1-kube-api-access-4t9qt\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-sbktn\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.318134 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.318063    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:18.588043 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.588015    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"]
Sep 16 17:10:18.590386 ip-10-0-143-143 kubenswrapper[2574]: W0916 17:10:18.590360    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92133754_268b_4699_b65b_23af1d99fbd1.slice/crio-da1be285cc437570a9ec26cb17ce29152b9e0f763e8bf23026e3cfd4850ad641 WatchSource:0}: Error finding container da1be285cc437570a9ec26cb17ce29152b9e0f763e8bf23026e3cfd4850ad641: Status 404 returned error can't find the container with id da1be285cc437570a9ec26cb17ce29152b9e0f763e8bf23026e3cfd4850ad641
Sep 16 17:10:18.810880 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.810840    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn" event={"ID":"92133754-268b-4699-b65b-23af1d99fbd1","Type":"ContainerStarted","Data":"0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6"}
Sep 16 17:10:18.811087 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.810907    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn" event={"ID":"92133754-268b-4699-b65b-23af1d99fbd1","Type":"ContainerStarted","Data":"da1be285cc437570a9ec26cb17ce29152b9e0f763e8bf23026e3cfd4850ad641"}
Sep 16 17:10:18.830997 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:18.830948    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn" podStartSLOduration=1.830932062 podStartE2EDuration="1.830932062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:10:18.82846434 +0000 UTC m=+4815.348087116" watchObservedRunningTime="2026-09-16 17:10:18.830932062 +0000 UTC m=+4815.350554836"
Sep 16 17:10:19.318825 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:19.318786    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:26.910021 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:26.909962    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"]
Sep 16 17:10:26.920738 ip-10-0-143-143 kubenswrapper[2574]: E0916 17:10:26.920698    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:10:26.920738 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:10:26.920738 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 17:10:26.920738 ip-10-0-143-143 kubenswrapper[2574]:  > containerID="0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6" cmd=["/bin/sleep","15"]
Sep 16 17:10:26.920983 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:26.920755    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn" message="" err=<
Sep 16 17:10:26.920983 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:10:26.920983 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 17:10:26.920983 ip-10-0-143-143 kubenswrapper[2574]:  >
Sep 16 17:10:26.920983 ip-10-0-143-143 kubenswrapper[2574]: E0916 17:10:26.920768    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:10:26.920983 ip-10-0-143-143 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:10:26.920983 ip-10-0-143-143 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 17:10:26.920983 ip-10-0-143-143 kubenswrapper[2574]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn" podUID="92133754-268b-4699-b65b-23af1d99fbd1" containerName="main" containerID="cri-o://0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6"
Sep 16 17:10:26.920983 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:26.920805    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn" podUID="92133754-268b-4699-b65b-23af1d99fbd1" containerName="main" containerID="cri-o://0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6" gracePeriod=60
Sep 16 17:10:27.166142 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.166077    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:27.308739 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.308701    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92133754-268b-4699-b65b-23af1d99fbd1-tls-certs\") pod \"92133754-268b-4699-b65b-23af1d99fbd1\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") "
Sep 16 17:10:27.308739 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.308756    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-dshm\") pod \"92133754-268b-4699-b65b-23af1d99fbd1\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") "
Sep 16 17:10:27.309021 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.308851    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-model-cache\") pod \"92133754-268b-4699-b65b-23af1d99fbd1\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") "
Sep 16 17:10:27.309021 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.308885    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-home\") pod \"92133754-268b-4699-b65b-23af1d99fbd1\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") "
Sep 16 17:10:27.309021 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.308917    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t9qt\" (UniqueName: \"kubernetes.io/projected/92133754-268b-4699-b65b-23af1d99fbd1-kube-api-access-4t9qt\") pod \"92133754-268b-4699-b65b-23af1d99fbd1\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") "
Sep 16 17:10:27.309021 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.308974    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-tmp-dir\") pod \"92133754-268b-4699-b65b-23af1d99fbd1\" (UID: \"92133754-268b-4699-b65b-23af1d99fbd1\") "
Sep 16 17:10:27.309431 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.309215    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-model-cache" (OuterVolumeSpecName: "model-cache") pod "92133754-268b-4699-b65b-23af1d99fbd1" (UID: "92133754-268b-4699-b65b-23af1d99fbd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:10:27.309503 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.309474    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92133754-268b-4699-b65b-23af1d99fbd1" (UID: "92133754-268b-4699-b65b-23af1d99fbd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:10:27.309589 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.309494    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-home" (OuterVolumeSpecName: "home") pod "92133754-268b-4699-b65b-23af1d99fbd1" (UID: "92133754-268b-4699-b65b-23af1d99fbd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:10:27.311664 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.311631    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-dshm" (OuterVolumeSpecName: "dshm") pod "92133754-268b-4699-b65b-23af1d99fbd1" (UID: "92133754-268b-4699-b65b-23af1d99fbd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:10:27.311764 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.311667    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92133754-268b-4699-b65b-23af1d99fbd1-kube-api-access-4t9qt" (OuterVolumeSpecName: "kube-api-access-4t9qt") pod "92133754-268b-4699-b65b-23af1d99fbd1" (UID: "92133754-268b-4699-b65b-23af1d99fbd1"). InnerVolumeSpecName "kube-api-access-4t9qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:10:27.311825 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.311770    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92133754-268b-4699-b65b-23af1d99fbd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92133754-268b-4699-b65b-23af1d99fbd1" (UID: "92133754-268b-4699-b65b-23af1d99fbd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:10:27.410544 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.410481    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9qt\" (UniqueName: \"kubernetes.io/projected/92133754-268b-4699-b65b-23af1d99fbd1-kube-api-access-4t9qt\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:10:27.410544 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.410511    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-tmp-dir\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:10:27.410544 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.410543    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92133754-268b-4699-b65b-23af1d99fbd1-tls-certs\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:10:27.410544 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.410552    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-dshm\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:10:27.410815 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.410561    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-model-cache\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:10:27.410815 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.410569    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92133754-268b-4699-b65b-23af1d99fbd1-home\") on node \"ip-10-0-143-143.ec2.internal\" DevicePath \"\""
Sep 16 17:10:27.854579 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.854543    2574 generic.go:356] "Generic (PLEG): container finished" podID="92133754-268b-4699-b65b-23af1d99fbd1" containerID="0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6" exitCode=0
Sep 16 17:10:27.854797 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.854640    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"
Sep 16 17:10:27.854797 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.854662    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn" event={"ID":"92133754-268b-4699-b65b-23af1d99fbd1","Type":"ContainerDied","Data":"0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6"}
Sep 16 17:10:27.854797 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.854718    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn" event={"ID":"92133754-268b-4699-b65b-23af1d99fbd1","Type":"ContainerDied","Data":"da1be285cc437570a9ec26cb17ce29152b9e0f763e8bf23026e3cfd4850ad641"}
Sep 16 17:10:27.854797 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.854748    2574 scope.go:117] "RemoveContainer" containerID="0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6"
Sep 16 17:10:27.864307 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.864286    2574 scope.go:117] "RemoveContainer" containerID="0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6"
Sep 16 17:10:27.864603 ip-10-0-143-143 kubenswrapper[2574]: E0916 17:10:27.864578    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6\": container with ID starting with 0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6 not found: ID does not exist" containerID="0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6"
Sep 16 17:10:27.864684 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.864612    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6"} err="failed to get container status \"0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6\": rpc error: code = NotFound desc = could not find container \"0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6\": container with ID starting with 0ba43f0f9d138088bd800aab083e4ade5b7d02cbd287e7d8a531ea03512e6eb6 not found: ID does not exist"
Sep 16 17:10:27.874574 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.874548    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"]
Sep 16 17:10:27.878694 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.878665    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-sbktn"]
Sep 16 17:10:27.995085 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:10:27.995050    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92133754-268b-4699-b65b-23af1d99fbd1" path="/var/lib/kubelet/pods/92133754-268b-4699-b65b-23af1d99fbd1/volumes"
Sep 16 17:11:39.271506 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:39.271452    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9582c_f9980823-1d5e-4996-8eec-7054d9091b82/istio-proxy/0.log"
Sep 16 17:11:39.301400 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:39.301363    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9582c_f9980823-1d5e-4996-8eec-7054d9091b82/istio-proxy/1.log"
Sep 16 17:11:39.315623 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:39.315587    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79955bb76b-kbcfl_3b6415b2-6e76-454e-b85e-4e7e80cbd01e/router/0.log"
Sep 16 17:11:40.028585 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:40.028549    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9582c_f9980823-1d5e-4996-8eec-7054d9091b82/istio-proxy/0.log"
Sep 16 17:11:40.056617 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:40.056587    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9582c_f9980823-1d5e-4996-8eec-7054d9091b82/istio-proxy/1.log"
Sep 16 17:11:40.070041 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:40.069991    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79955bb76b-kbcfl_3b6415b2-6e76-454e-b85e-4e7e80cbd01e/router/0.log"
Sep 16 17:11:41.277769 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:41.277736    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wdgxz_5288fd6d-8a27-4941-917c-f8cf29e38b2e/manager/0.log"
Sep 16 17:11:47.228785 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.228753    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zgcmz/must-gather-n4gkv"]
Sep 16 17:11:47.229250 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.229130    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92133754-268b-4699-b65b-23af1d99fbd1" containerName="main"
Sep 16 17:11:47.229250 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.229143    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="92133754-268b-4699-b65b-23af1d99fbd1" containerName="main"
Sep 16 17:11:47.229250 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.229205    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="92133754-268b-4699-b65b-23af1d99fbd1" containerName="main"
Sep 16 17:11:47.232367 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.232347    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgcmz/must-gather-n4gkv"
Sep 16 17:11:47.234756 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.234726    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zgcmz\"/\"kube-root-ca.crt\""
Sep 16 17:11:47.234886 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.234795    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zgcmz\"/\"openshift-service-ca.crt\""
Sep 16 17:11:47.235351 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.235334    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zgcmz\"/\"default-dockercfg-lmlsv\""
Sep 16 17:11:47.241140 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.241116    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgcmz/must-gather-n4gkv"]
Sep 16 17:11:47.326430 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.326388    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m92wm\" (UniqueName: \"kubernetes.io/projected/095e3aa5-9fb0-4640-87a3-6450e33eaedf-kube-api-access-m92wm\") pod \"must-gather-n4gkv\" (UID: \"095e3aa5-9fb0-4640-87a3-6450e33eaedf\") " pod="openshift-must-gather-zgcmz/must-gather-n4gkv"
Sep 16 17:11:47.326633 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.326439    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/095e3aa5-9fb0-4640-87a3-6450e33eaedf-must-gather-output\") pod \"must-gather-n4gkv\" (UID: \"095e3aa5-9fb0-4640-87a3-6450e33eaedf\") " pod="openshift-must-gather-zgcmz/must-gather-n4gkv"
Sep 16 17:11:47.427565 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.427504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/095e3aa5-9fb0-4640-87a3-6450e33eaedf-must-gather-output\") pod \"must-gather-n4gkv\" (UID: \"095e3aa5-9fb0-4640-87a3-6450e33eaedf\") " pod="openshift-must-gather-zgcmz/must-gather-n4gkv"
Sep 16 17:11:47.427770 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.427636    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m92wm\" (UniqueName: \"kubernetes.io/projected/095e3aa5-9fb0-4640-87a3-6450e33eaedf-kube-api-access-m92wm\") pod \"must-gather-n4gkv\" (UID: \"095e3aa5-9fb0-4640-87a3-6450e33eaedf\") " pod="openshift-must-gather-zgcmz/must-gather-n4gkv"
Sep 16 17:11:47.427945 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.427920    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/095e3aa5-9fb0-4640-87a3-6450e33eaedf-must-gather-output\") pod \"must-gather-n4gkv\" (UID: \"095e3aa5-9fb0-4640-87a3-6450e33eaedf\") " pod="openshift-must-gather-zgcmz/must-gather-n4gkv"
Sep 16 17:11:47.435489 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.435462    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m92wm\" (UniqueName: \"kubernetes.io/projected/095e3aa5-9fb0-4640-87a3-6450e33eaedf-kube-api-access-m92wm\") pod \"must-gather-n4gkv\" (UID: \"095e3aa5-9fb0-4640-87a3-6450e33eaedf\") " pod="openshift-must-gather-zgcmz/must-gather-n4gkv"
Sep 16 17:11:47.542368 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.542274    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgcmz/must-gather-n4gkv"
Sep 16 17:11:47.812169 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.812141    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgcmz/must-gather-n4gkv"]
Sep 16 17:11:47.814415 ip-10-0-143-143 kubenswrapper[2574]: W0916 17:11:47.814386    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod095e3aa5_9fb0_4640_87a3_6450e33eaedf.slice/crio-8ded495d665a4edd32266c6d47dc7972372e5076f8b2f3d83585174a01db6f24 WatchSource:0}: Error finding container 8ded495d665a4edd32266c6d47dc7972372e5076f8b2f3d83585174a01db6f24: Status 404 returned error can't find the container with id 8ded495d665a4edd32266c6d47dc7972372e5076f8b2f3d83585174a01db6f24
Sep 16 17:11:47.816558 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:47.816533    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:11:48.230936 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:48.230895    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgcmz/must-gather-n4gkv" event={"ID":"095e3aa5-9fb0-4640-87a3-6450e33eaedf","Type":"ContainerStarted","Data":"8ded495d665a4edd32266c6d47dc7972372e5076f8b2f3d83585174a01db6f24"}
Sep 16 17:11:49.240731 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:49.240691    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgcmz/must-gather-n4gkv" event={"ID":"095e3aa5-9fb0-4640-87a3-6450e33eaedf","Type":"ContainerStarted","Data":"676480462c25121ef3412b358684253aa43b498777f13fa697bc9a5085c798b4"}
Sep 16 17:11:49.241109 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:49.240739    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgcmz/must-gather-n4gkv" event={"ID":"095e3aa5-9fb0-4640-87a3-6450e33eaedf","Type":"ContainerStarted","Data":"a58fdbf745e3fc3e64db4ab3810b6c4ce7fb8f8213c8e063f853182c0ba799a4"}
Sep 16 17:11:49.267201 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:49.267136    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zgcmz/must-gather-n4gkv" podStartSLOduration=1.381176954 podStartE2EDuration="2.267111835s" totalImagesPullingTime="885.934881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:11:49.256507027 +0000 UTC m=+4905.776129803" watchObservedRunningTime="2026-09-16 17:11:49.267111835 +0000 UTC m=+4905.786734612"
Sep 16 17:11:50.241110 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:50.241076    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-86l22_9823bb5e-f7d2-442d-b83b-c833dafb42ba/global-pull-secret-syncer/0.log"
Sep 16 17:11:50.427828 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:50.427783    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s2k52_9a690936-8677-451d-8ff9-c6b3b92d666a/konnectivity-agent/0.log"
Sep 16 17:11:50.502137 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:50.502048    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-143.ec2.internal_ed97d765e2c9a80c45b1435ad72f4818/haproxy/0.log"
Sep 16 17:11:56.114247 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:56.114212    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wdgxz_5288fd6d-8a27-4941-917c-f8cf29e38b2e/manager/0.log"
Sep 16 17:11:57.180308 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.180271    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23/alertmanager/0.log"
Sep 16 17:11:57.206706 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.206669    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23/config-reloader/0.log"
Sep 16 17:11:57.233400 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.233365    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23/kube-rbac-proxy-web/0.log"
Sep 16 17:11:57.266864 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.266780    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23/kube-rbac-proxy/0.log"
Sep 16 17:11:57.290333 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.290302    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23/kube-rbac-proxy-metric/0.log"
Sep 16 17:11:57.337597 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.336999    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23/prom-label-proxy/0.log"
Sep 16 17:11:57.364321 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.364282    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1718f3eb-dd98-40a9-bab4-d3b1f4fb0a23/init-config-reloader/0.log"
Sep 16 17:11:57.417805 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.417475    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-28r4q_591b0e66-93df-447a-8ddd-d850e006a2a5/cluster-monitoring-operator/0.log"
Sep 16 17:11:57.547795 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.547765    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-66568f976d-rkd2f_6962731d-8a32-42c9-8cac-e137bef90b68/metrics-server/0.log"
Sep 16 17:11:57.787115 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.787022    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qslcd_38e7589c-0441-4224-a5f1-e80d4759d220/node-exporter/0.log"
Sep 16 17:11:57.812827 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.812722    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qslcd_38e7589c-0441-4224-a5f1-e80d4759d220/kube-rbac-proxy/0.log"
Sep 16 17:11:57.838125 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.838094    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qslcd_38e7589c-0441-4224-a5f1-e80d4759d220/init-textfile/0.log"
Sep 16 17:11:57.873223 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.873184    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-98x75_e72a7c5c-e549-45e7-8c91-f1d04a832452/kube-rbac-proxy-main/0.log"
Sep 16 17:11:57.894371 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.894342    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-98x75_e72a7c5c-e549-45e7-8c91-f1d04a832452/kube-rbac-proxy-self/0.log"
Sep 16 17:11:57.915961 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.915926    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-98x75_e72a7c5c-e549-45e7-8c91-f1d04a832452/openshift-state-metrics/0.log"
Sep 16 17:11:57.956216 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.956180    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c7548b24-c2b9-4da6-a35d-8ba81ee1086c/prometheus/0.log"
Sep 16 17:11:57.974149 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:57.974105    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c7548b24-c2b9-4da6-a35d-8ba81ee1086c/config-reloader/0.log"
Sep 16 17:11:58.001872 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:58.001842    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c7548b24-c2b9-4da6-a35d-8ba81ee1086c/thanos-sidecar/0.log"
Sep 16 17:11:58.028505 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:58.028437    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c7548b24-c2b9-4da6-a35d-8ba81ee1086c/kube-rbac-proxy-web/0.log"
Sep 16 17:11:58.058195 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:58.058114    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c7548b24-c2b9-4da6-a35d-8ba81ee1086c/kube-rbac-proxy/0.log"
Sep 16 17:11:58.086326 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:58.086293    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c7548b24-c2b9-4da6-a35d-8ba81ee1086c/kube-rbac-proxy-thanos/0.log"
Sep 16 17:11:58.108875 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:58.108842    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c7548b24-c2b9-4da6-a35d-8ba81ee1086c/init-config-reloader/0.log"
Sep 16 17:11:58.217174 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:58.217141    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6786f7c4c4-8srfn_86294e16-98ef-4a59-aae0-68600f10abbf/telemeter-client/0.log"
Sep 16 17:11:58.240335 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:58.240304    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6786f7c4c4-8srfn_86294e16-98ef-4a59-aae0-68600f10abbf/reload/0.log"
Sep 16 17:11:58.262366 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:58.262335    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6786f7c4c4-8srfn_86294e16-98ef-4a59-aae0-68600f10abbf/kube-rbac-proxy/0.log"
Sep 16 17:11:58.297208 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:58.297172    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57cb8f89c6-7h58l_2781fc36-78d3-4e86-b25c-6e99a11328fb/thanos-query/0.log"
Sep 16 17:11:58.321234 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:58.321158    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57cb8f89c6-7h58l_2781fc36-78d3-4e86-b25c-6e99a11328fb/kube-rbac-proxy-web/0.log"
Sep 16 17:11:58.349707 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:58.349674    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57cb8f89c6-7h58l_2781fc36-78d3-4e86-b25c-6e99a11328fb/kube-rbac-proxy/0.log"
Sep 16 17:11:58.370928 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:58.370894    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57cb8f89c6-7h58l_2781fc36-78d3-4e86-b25c-6e99a11328fb/prom-label-proxy/0.log"
Sep 16 17:11:58.392716 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:58.392683    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57cb8f89c6-7h58l_2781fc36-78d3-4e86-b25c-6e99a11328fb/kube-rbac-proxy-rules/0.log"
Sep 16 17:11:58.412574 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:11:58.412541    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57cb8f89c6-7h58l_2781fc36-78d3-4e86-b25c-6e99a11328fb/kube-rbac-proxy-metrics/0.log"
Sep 16 17:12:00.056359 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.056321    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"]
Sep 16 17:12:00.061721 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.061688    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.069803 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.069774    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a9726de-16f7-4df4-a936-ff9a8e8371cf-sys\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.070053 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.070031    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a9726de-16f7-4df4-a936-ff9a8e8371cf-lib-modules\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.070244 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.070222    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a9726de-16f7-4df4-a936-ff9a8e8371cf-podres\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.070335 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.070277    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a9726de-16f7-4df4-a936-ff9a8e8371cf-proc\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.070335 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.070308    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwjr7\" (UniqueName: \"kubernetes.io/projected/2a9726de-16f7-4df4-a936-ff9a8e8371cf-kube-api-access-dwjr7\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.070881 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.070859    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"]
Sep 16 17:12:00.171862 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.171815    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a9726de-16f7-4df4-a936-ff9a8e8371cf-lib-modules\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.172048 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.171917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a9726de-16f7-4df4-a936-ff9a8e8371cf-podres\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.172048 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.171944    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a9726de-16f7-4df4-a936-ff9a8e8371cf-proc\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.172048 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.171970    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjr7\" (UniqueName: \"kubernetes.io/projected/2a9726de-16f7-4df4-a936-ff9a8e8371cf-kube-api-access-dwjr7\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.172048 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.172002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a9726de-16f7-4df4-a936-ff9a8e8371cf-sys\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.172048 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.172022    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a9726de-16f7-4df4-a936-ff9a8e8371cf-lib-modules\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.172305 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.172048    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a9726de-16f7-4df4-a936-ff9a8e8371cf-proc\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.172305 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.172081    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a9726de-16f7-4df4-a936-ff9a8e8371cf-podres\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.172305 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.172099    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a9726de-16f7-4df4-a936-ff9a8e8371cf-sys\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.180488 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.180461    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjr7\" (UniqueName: \"kubernetes.io/projected/2a9726de-16f7-4df4-a936-ff9a8e8371cf-kube-api-access-dwjr7\") pod \"perf-node-gather-daemonset-p82hs\" (UID: \"2a9726de-16f7-4df4-a936-ff9a8e8371cf\") " pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.374301 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.374266    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:00.613198 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.613165    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-979kp_15e1a31e-0ac4-4e41-9d1a-be65065d6089/download-server/0.log"
Sep 16 17:12:00.671755 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:00.671706    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"]
Sep 16 17:12:00.673864 ip-10-0-143-143 kubenswrapper[2574]: W0916 17:12:00.673827    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2a9726de_16f7_4df4_a936_ff9a8e8371cf.slice/crio-d03046795c5a5516e17b8b2cd63118a3b4a14a78738bc55e47d6a55d07a5f404 WatchSource:0}: Error finding container d03046795c5a5516e17b8b2cd63118a3b4a14a78738bc55e47d6a55d07a5f404: Status 404 returned error can't find the container with id d03046795c5a5516e17b8b2cd63118a3b4a14a78738bc55e47d6a55d07a5f404
Sep 16 17:12:01.321666 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:01.321621    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs" event={"ID":"2a9726de-16f7-4df4-a936-ff9a8e8371cf","Type":"ContainerStarted","Data":"34a2472009451587bf962f28a7cbdb301dae5b7c5a34a3242e42f2266fa4d10e"}
Sep 16 17:12:01.322039 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:01.321699    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:01.322039 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:01.321726    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs" event={"ID":"2a9726de-16f7-4df4-a936-ff9a8e8371cf","Type":"ContainerStarted","Data":"d03046795c5a5516e17b8b2cd63118a3b4a14a78738bc55e47d6a55d07a5f404"}
Sep 16 17:12:01.338957 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:01.338905    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs" podStartSLOduration=1.338886949 podStartE2EDuration="1.338886949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:12:01.337122402 +0000 UTC m=+4917.856745190" watchObservedRunningTime="2026-09-16 17:12:01.338886949 +0000 UTC m=+4917.858509781"
Sep 16 17:12:02.422695 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:02.422649    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wq8bz_cb25687b-a293-40ae-a3a7-e2357384bcd4/dns/0.log"
Sep 16 17:12:02.447279 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:02.447249    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wq8bz_cb25687b-a293-40ae-a3a7-e2357384bcd4/kube-rbac-proxy/0.log"
Sep 16 17:12:02.473761 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:02.473734    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6r28d_eedc6ccd-a007-4ca4-98ba-54fa62367075/dns-node-resolver/0.log"
Sep 16 17:12:03.114317 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:03.114283    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nz7nc_12c61b9c-b819-4330-bcf8-05341ce39ac8/node-ca/0.log"
Sep 16 17:12:04.242396 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:04.242363    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9582c_f9980823-1d5e-4996-8eec-7054d9091b82/istio-proxy/0.log"
Sep 16 17:12:04.271436 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:04.271411    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9582c_f9980823-1d5e-4996-8eec-7054d9091b82/istio-proxy/1.log"
Sep 16 17:12:04.295325 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:04.295300    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79955bb76b-kbcfl_3b6415b2-6e76-454e-b85e-4e7e80cbd01e/router/0.log"
Sep 16 17:12:04.875216 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:04.875175    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-twc8n_1d1a3f5c-a7cc-45cd-8ff2-e04f658cdb59/serve-healthcheck-canary/0.log"
Sep 16 17:12:05.331490 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:05.331401    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-whb7g_29cf843e-24ce-4d87-baea-993e52f3d2e4/insights-operator/0.log"
Sep 16 17:12:05.332915 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:05.332888    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-whb7g_29cf843e-24ce-4d87-baea-993e52f3d2e4/insights-operator/1.log"
Sep 16 17:12:05.494221 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:05.494192    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c6rkl_acac2a59-951e-427a-96ee-52e273629779/kube-rbac-proxy/0.log"
Sep 16 17:12:05.514870 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:05.514843    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c6rkl_acac2a59-951e-427a-96ee-52e273629779/exporter/0.log"
Sep 16 17:12:05.538116 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:05.538086    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c6rkl_acac2a59-951e-427a-96ee-52e273629779/extractor/0.log"
Sep 16 17:12:07.337352 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:07.337320    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zgcmz/perf-node-gather-daemonset-p82hs"
Sep 16 17:12:08.453971 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:08.453937    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-75c8bf98c8-dvlgl_a2b5ce5e-09a1-450f-8636-86e76195e6f5/manager/0.log"
Sep 16 17:12:09.556226 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:09.556187    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9582c_f9980823-1d5e-4996-8eec-7054d9091b82/istio-proxy/0.log"
Sep 16 17:12:09.608454 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:09.608416    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9582c_f9980823-1d5e-4996-8eec-7054d9091b82/istio-proxy/1.log"
Sep 16 17:12:09.629363 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:09.629296    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79955bb76b-kbcfl_3b6415b2-6e76-454e-b85e-4e7e80cbd01e/router/0.log"
Sep 16 17:12:10.020540 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:10.020305    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6c85c996cc-nw4bw_a194273b-a5f5-4ab2-a47f-3e87dc940b54/manager/0.log"
Sep 16 17:12:10.045231 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:10.045193    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-wx6zr_39d33713-d0a3-4ea9-b6aa-273f0ee0d9f9/server/0.log"
Sep 16 17:12:10.948782 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:10.948720    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9582c_f9980823-1d5e-4996-8eec-7054d9091b82/istio-proxy/0.log"
Sep 16 17:12:10.981990 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:10.981937    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9582c_f9980823-1d5e-4996-8eec-7054d9091b82/istio-proxy/1.log"
Sep 16 17:12:10.999984 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:10.999949    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79955bb76b-kbcfl_3b6415b2-6e76-454e-b85e-4e7e80cbd01e/router/0.log"
Sep 16 17:12:11.111916 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:11.111883    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-4wp2q_9e7515de-5f8e-4760-aee4-20245bfa806e/seaweedfs/0.log"
Sep 16 17:12:16.017004 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:16.016973    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-kglsn_0c163fdb-43f8-4589-bf57-c156d14ad5ca/migrator/0.log"
Sep 16 17:12:16.036540 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:16.036494    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-kglsn_0c163fdb-43f8-4589-bf57-c156d14ad5ca/graceful-termination/0.log"
Sep 16 17:12:17.404090 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:17.404051    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7jbf9_c685b2f9-18ae-4c7f-b9c7-8f271c676488/kube-multus/0.log"
Sep 16 17:12:17.432288 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:17.432262    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2w8vv_9fe8739d-6acf-41c1-b1c8-f11a8fa361ca/kube-multus-additional-cni-plugins/0.log"
Sep 16 17:12:17.455117 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:17.455081    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2w8vv_9fe8739d-6acf-41c1-b1c8-f11a8fa361ca/egress-router-binary-copy/0.log"
Sep 16 17:12:17.486931 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:17.486898    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2w8vv_9fe8739d-6acf-41c1-b1c8-f11a8fa361ca/cni-plugins/0.log"
Sep 16 17:12:17.508938 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:17.508905    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2w8vv_9fe8739d-6acf-41c1-b1c8-f11a8fa361ca/bond-cni-plugin/0.log"
Sep 16 17:12:17.528711 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:17.528682    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2w8vv_9fe8739d-6acf-41c1-b1c8-f11a8fa361ca/routeoverride-cni/0.log"
Sep 16 17:12:17.550668 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:17.550642    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2w8vv_9fe8739d-6acf-41c1-b1c8-f11a8fa361ca/whereabouts-cni-bincopy/0.log"
Sep 16 17:12:17.574003 ip-10-0-143-143 kubenswrapper[2574]: I0916 17:12:17.573951    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2w8vv_9fe8739d-6acf-41c1-b1c8-f11a8fa361ca/whereabouts-cni/0.log"